by Duncan Epping
Just noticed the following topic, which definitely contains some good info about how ESX deals with memory. Thanks to Kit for clearing things up. These posts are valuable, keep ’em coming!
a short outtake:
Basically how aggressive do you want TPS to scan for shared pages? Obviously TPS has a cost to running in terms of CPU, but has benefits in terms of reduced memory usage. So there’s a tradeoff. We have a default that we think is a good balance, but we let the user modify that if they want.
At a customer site we noticed that the ESX hosts were swapping, Nagios generated a nice alarm. After some research it seemed like certain VM’s were swapping to the VMFS volume, so not inside the OS but VMware swap usage. A closer look at the system revealed that we weren’t overcommitting. There was over 6GB of memory free and there were no limit’s set to the specific VM. Could it be just Nagios or… No, esxtop with the following commands “s2
The column swcur displays the current swap file usage, I marked the values higher than 0 red.
After a couple of searches it seemed that there is little info about swcur. But Kit Colbert, a VMware employee, posted on the vmtn forum about checking your current memory / swap usage in the file “/proc/vmware/sched/mem”. With cat you can easily display this, and with “watch -n 1” you can refresh your view every second. The following output was retrieved via the command “watch -n 1 cat /proc/vmware/sched/mem”:
We’ve migrated a VM which was swapping according to esxtop and nagios to another host, and as expected the swap remained. We powered down a VM that was swapping, and although the host had more than enough free mem available, the swap returned. It was less than before but still… The funny thing is that according to Kit it’s all about the column “swap out” and we did not see much action going on there.
I’m dazzled, anyone?
Yesterday evening I witnessed a weird phenomenon. We had to bring down a complete environment to move a 19″ rack to a different location. We switched the SAN on, waited a couple of minutes and switched the ESX hosts on. When the ESX hosts finished booting we booted the VirtualCenter. Everything looked normal in the VI Client. I had all connections to the SAN and all ESX Hosts were up and running. So I decided to power up the first VM, it was a VM named LNX01. Within a second the VM got renamed to LNX05(1). I though I was going nuts. I checked the settings of the renamed VM and indeed it was pointing out to the LNX05 diskfiles/vmx.
Maybe it was just me, or this one VM so I decided to give another one a try, I powered up LNX02. Same happening here, within a second the VM was renamed to PS01(1) and booted fine. The settings were pointing out to PS01. I checked a couple of VM’s but could not find anything weird. I restarted the VirtualCenter service just to be sure. I started the VM LNX03 and again it was renamed… Than I decided to restart the “mgmt-vmware” services on all of the ESX hosts and the problem never returned again. It seems like VirtualCenter had a different view than the ESX hosts had. But I can’t think of a logical reason what could cause this. I searched the knowledge base but could not find any related problems, well besides an old article based on VirtualCenter 1.2.
Today a customer called about a problem with the Exchange VM. For some reason the ESX Host where this VM resided was always swapping/ballooning. They checked and double checked the settings but could not find the problem. After a quick scan I noticed that there were limits set on memory for each VM. This particular VM had 1536MB of memory and a limit of 1024MB. After changing the setting back to it’s default setting, “unlimited”, the message was gone. I haven’t got a clue why this setup this way, limitting the memory to the exact amount assigned to a VM… weird, but problem solved.
I just received the following:
New patches are available for ESX Server 3.0.1 and ESX Server 3.0.2.
ESX Server patches can be accessed through the on-line ESX Server Patch Download tool:
[http://app.connect.vmware.com/e/er.aspx?s=524&lid=2272&elq=4D5713F997F04342981A231DF09609D0“>]Please follow the instructions on the appropriate patch download page.
VMware ESX Server 3.0.1 Patches
ESX-1003516 (Critical): fixes an issue where snapshot of a Suspended Virtual Machine fails;includes a fix for Windows OEM CDs Installation Failure; fixes an issue where Virtual Machines stop responding When Updated with Symantec Virus Definition
ESX-1003517 (Security):updates the e2fsprogs package
ESX-1003518 (Critical):changes the default Multipath Policy;fixes an issue where ESX Server stops responding due to low COW Heap Memory;fixes an issue where ESX Server hosts stop responding due to a spinlock timeout;includes fixes for SysAlert Warning Message, Partially Disabled Processors
ESX-1003520 (General):fixes an issue with the USB Subsystem and High-Speed USB HID Devices
ESX-1003521 (Security):Updates the libxml2 UtilityVMware ESX Server 3.0.2 Patches
ESX-1003523 (Security):updates the e2fsprogs package
ESX-1003524 (Critical):updates VMware-esx-vmkernel, VMware-esx-vmx, and VMware-esx-apps
ESX-1003526 (General):fixes an issue with USB Subsystem and High-Speed USB HID
ESX-1003527 (General):adds support for Ubuntu 7.10 GA
ESX-1003528 (Security):updates the libxml2 UtilityWe expect the next patch release in late April 2008. If you have any questions, please contact support at 877-4-VMWARE.