Yellow Bricks

by Duncan Epping

Swapping, esxtop and /proc/vmware/sched/mem

· ·

At a customer site we noticed that the ESX hosts were swapping, Nagios generated a nice alarm. After some research it seemed like certain VM’s were swapping to the VMFS volume, so not inside the OS but VMware swap usage. A closer look at the system revealed that we weren’t overcommitting. There was over 6GB of memory free and there were no limit’s set to the specific VM. Could it be just Nagios or… No, esxtop with the following commands “s2 m f j” revealed the following:

The column swcur displays the current swap file usage, I marked the values higher than 0 red.

After a couple of searches it seemed that there is little info about swcur. But Kit Colbert, a VMware employee, posted on the vmtn forum about checking your current memory / swap usage in the file “/proc/vmware/sched/mem”. With cat you can easily display this, and with “watch -n 1” you can refresh your view every second. The following output was retrieved via the command “watch -n 1 cat /proc/vmware/sched/mem”:

We’ve migrated a VM which was swapping according to esxtop and nagios to another host, and as expected the swap remained. We powered down a VM that was swapping, and although the host had more than enough free mem available, the swap returned. It was less than before but still… The funny thing is that according to Kit it’s all about the column “swap out” and we did not see much action going on there.

I’m dazzled, anyone?

Reader Interactions

Comments

  1. Steven says

    Unfortunately I’m no help for actually answering your query, but out of curiosity what were you monitoring from Nagios that generated the alarm? Was it an OID or something from the VmPerl API, or something else?

  5. Dave Holland says

    Did you find anything?

    I’m also seeing this problem: on a 64GB ESX host, esxtop reports ~30GB free, but the host is using ~9GB swap (and has also pushed out ~8GB guest RAM using the balloon driver).

    Why is ESX swapping when there’s free RAM? There are no limits in use. Could misconfigured resource pools cause this?

  6. jmcole says

    We ultimately fixed this by installing/upgrading vmware tools after a kernel update & power cycle, giving all our VMs at least 25% of their declared ram as a reservation and powering off and powering on the VM one more time.

    I’d also suggested not putting on limits for cpu or ram on VMs or resource pools, if your available hardware allows. In addition make sure if you login to the VM via VIC console to make sure and log out each time.

    We first noticed the issue with timekeeping issues within our Linux VMs and tweaking per recommendations stated on: http://kb.vmware.com/kb/1006427 weren’t helping…in the end it was the ~27GB of swap we had flying around our fiber network that was the bottleneck.

    Now that vmware tools, specifically the memory manager is enabled, NTP is able to keep up very easily using the aforementioned tips on the KB article. Good luck!

    p.s. we are running esx 3.0.3 and vs 2.5 u4 and redhat/centos 5.x VMs.