At a customer site we noticed that the ESX hosts were swapping, Nagios generated a nice alarm. After some research it seemed like certain VM’s were swapping to the VMFS volume, so not inside the OS but VMware swap usage. A closer look at the system revealed that we weren’t overcommitting. There was over 6GB of memory free and there were no limit’s set to the specific VM. Could it be just Nagios or… No, esxtop with the following commands “s2
The column swcur displays the current swap file usage, I marked the values higher than 0 red.
After a couple of searches it seemed that there is little info about swcur. But Kit Colbert, a VMware employee, posted on the vmtn forum about checking your current memory / swap usage in the file “/proc/vmware/sched/mem”. With cat you can easily display this, and with “watch -n 1” you can refresh your view every second. The following output was retrieved via the command “watch -n 1 cat /proc/vmware/sched/mem”:
We’ve migrated a VM which was swapping according to esxtop and nagios to another host, and as expected the swap remained. We powered down a VM that was swapping, and although the host had more than enough free mem available, the swap returned. It was less than before but still… The funny thing is that according to Kit it’s all about the column “swap out” and we did not see much action going on there.
I’m dazzled, anyone?
Steven says
Unfortunately I’m no help for actually answering your query, but out of curiosity what were you monitoring from Nagios that generated the alarm? Was it an OID or something from the VmPerl API, or something else?
Alastair says
Was this on an ESX 3.5 farm?
Was it a VM that had been VMotioned using DRS?
If so check out the KB article at http://kb.vmware.com/kb/1003638
I mentioned it when it hurt me, http://demitasse.co.nz/wordpress2/?p=29
Al.
Gabrie says
Hi
Did you see this new KB article?
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004901
Gabrie
kjolivier says
I am seeing this now…was there any lessons learned you can share?
Dave Holland says
Did you find anything?
I’m also seeing this problem: on a 64GB ESX host, esxtop reports ~30GB free, but the host is using ~9GB swap (and has also pushed out ~8GB guest RAM using the balloon driver).
Why is ESX swapping when there’s free RAM? There are no limits in use. Could misconfigured resource pools cause this?
jmcole says
We ultimately fixed this by installing/upgrading vmware tools after a kernel update & power cycle, giving all our VMs at least 25% of their declared ram as a reservation and powering off and powering on the VM one more time.
I’d also suggested not putting on limits for cpu or ram on VMs or resource pools, if your available hardware allows. In addition make sure if you login to the VM via VIC console to make sure and log out each time.
We first noticed the issue with timekeeping issues within our Linux VMs and tweaking per recommendations stated on: http://kb.vmware.com/kb/1006427 weren’t helping…in the end it was the ~27GB of swap we had flying around our fiber network that was the bottleneck.
Now that vmware tools, specifically the memory manager is enabled, NTP is able to keep up very easily using the aforementioned tips on the KB article. Good luck!
p.s. we are running esx 3.0.3 and vs 2.5 u4 and redhat/centos 5.x VMs.
kopper says
I am having the same issue
I just posted to get some feedback
http://communities.vmware.com/thread/276650?tstart=0
How To Get Popular On Youtube says
Fantastic blog overall! Your article was very insightful. I have numerous pages bookmarked relating to this, but your blog will be top of the list! Thanks very much