At a customer site we noticed that the ESX hosts were swapping, Nagios generated a nice alarm. After some research it seemed like certain VM’s were swapping to the VMFS volume, so not inside the OS but VMware swap usage. A closer look at the system revealed that we weren’t overcommitting. There was over 6GB of memory free and there were no limit’s set to the specific VM. Could it be just Nagios or… No, esxtop with the following commands “s2 m f j” revealed the following:

The column swcur displays the current swap file usage, I marked the values higher than 0 red.

After a couple of searches it seemed that there is little info about swcur. But Kit Colbert, a VMware employee, posted on the vmtn forum about checking your current memory / swap usage in the file “/proc/vmware/sched/mem”. With cat you can easily display this, and with “watch -n 1″ you can refresh your view every second. The following output was retrieved via the command “watch -n 1 cat /proc/vmware/sched/mem”:

We’ve migrated a VM which was swapping according to esxtop and nagios to another host, and as expected the swap remained. We powered down a VM that was swapping, and although the host had more than enough free mem available, the swap returned. It was less than before but still… The funny thing is that according to Kit it’s all about the column “swap out” and we did not see much action going on there.

I’m dazzled, anyone?