Yellow Bricks

by Duncan Epping

ESX

Is this VM actively swapping? (helping @heiner_hardt)

· ·

On twitter @heiner_hardt asked for help with a performance related issue he was experiencing. As I am starting to appreciate esxtop more every single day and I really start to appreciate solving performance problems I decided to dive in to it.

After the initial couple of questions Heiner posted a screenshot:

Heiner highlighted (red outline) a couple of metrics which indicated swapping and ballooning as he pointed out with the text boxes. Although I can’t disagree that swapping and ballooning happened at some point in time I do disagree with the conclusion that this virtual machine is swapping. Lets break it down:

Global Statistics:

  • 1393 Free -> Currently 1393MB memory available
  • High State -> Hypervisor is not under memory pressure
  • SWAP /MB 146 Cur -> 146MB has been swapped
  • SWAP /MB 83 Target -> Target amount that needed to be swapped was 83MB
  • 0.00 r/s -> No reads from swap currently
  • 0.00 w/s -> No writes to swap currently

World Statistics:

  • MCTLSZ 1307.27 -> The amount of guest physical memory that has been reclaimed by the balloon driver is 1307.27MB
  • MCTLTGT 1307.27 -> The amount of guest physical memory to be kept in the balloon driver is 1307.27MB
  • SWCUR 146.61 -> The current amount of memory that has been swapped is 146.61.
  • SWTGT 83.75 -> The target amount of memory that needed to be swapped was 83.75MB

Now that we know what these metrics mean and what the associated values are we can easily draw a conclusion:

At one point the host has most likely been overcommitted. However currently there is no memory pressure (state = high (>6% free memory)) as there is 1393MB of memory available. The metric “swcur” seems to indicate that swapping has occurred” however currently the host is not actively reading from swap or actively writing to swap (0.00 r/s and 0.00 w/s).

If the host is not experiencing memory pressure why is the balloon driver still inflated (MCTLTGT 1307.27MB)? Although the host is currently in a high memory state the amount of available memory almost equals the amount of claimed memory by the balloon driver. However deflating the balloon would return the host to a memory constrained state again.

My recommendation? Cut down on memory on your VMs! The fact that memory has been granted does not necessarily mean it is actively used and in this case it leads to serious overcommitment which in its turn leads to ballooning and even worse swapping.

One thing to point out though is the amount of “PSHARE” (TPS) is compared to average environments low. Might be something to explore!

VLAN ID 4095

· ·

One of my colleagues today asked me if it was possible to use VLAN ID 4095 for the “management” network of ESXi. This VLAN ID is however reserved for a very specific purpose.

This particular VLAN ID is only to be used for “Virtual Guest Tagging” (VGT). It basically means that the VLAN ID is stripped off at the Guest OS layer and not at the portgroup layer. In other words the VLAN trunk(multiple VLANs on a single wire) is extended to the virtual machine and the virtual machine will need to deal with it.

When will you use this? To be honest there aren’t many use cases any more. In the past it was used to increase the number of VLANs for a VM. The limit of 4 NICs for VI3 meant a maximum of 4 portgroups / VLANs per VM. However with vSphere the maximum amount of NICs went up to 10 and as such the amount of VLANs for a single VM also went up to 10.

Before people start to get excited about Virtual Guest Tagging, I personally prefer to stay away it. It heavily complicates the configuration of the VM and the vSwitch/dvSwitch and adds additional unneeded “stress” on your VMs vCPU.

PVSCSI and a 64bit OS

· ·

Yesterday we had an internal discussion about the support of PVSCSI in combination with a 64bit OS. VMware’s documentation currently states the following:

Paravirtual SCSI adapters are supported on the following guest operating systems:

Windows Server 2008
Windows Server 2003
Red Hat Enterprise Linux (RHEL) 5

source

As we normally spell out every single detail this KB article is kind of ambiguous in my opinion. To clarify it, both 32bit and 64bit versions of the detailed operating systems are currently supported (vSphere 4.0). One thing to note though is that there are still limitations, for instance booting a Linux guest from a disk attached to a PVSCSI adapter is currently not supported.

Swapping?

· ·

We had a discussion internally about performance and swapping. I started writing this article and asked Frank if it made sense. Frank’s reply “just guess what I am writing about at the moment”. As both of us had a different approach we decided to launch both articles at the same time and refer to each others post. So here’s the link to Frank’s take on the discussion and I highly recommend reading it: “Re: Swapping“.

As always the common theme of the discussion was “swapping bad”. Although I don’t necessarily disagree. I do want to note that it is important to figure out if the system is actually actively swapping or not.

In many cases “bad performance” is blamed on swapping. However this is not always the case. As described in my section on “ESXTOP”  there are multiple metrics on “swap” itself. Only a few of those relate to performance degradation due to swapping. I’ve listed the important metrics below.

Host:
MEM – SWAP/MB curr = Total swapped machine memory of all the groups including virtual machines.
MEM – SWAP/MB “target” = The expected swap usage.
MEM – SWAP/MB “r/s” = The rate at which memory is swapped in from disk.
MEM – SWAP/MB “w/s” = the rate at machine memory is swapped out to disk.

VM:
MEM – SWCUR = If larger than 0 host has swapped memory pages from this VM in the past.
MEM – SWTGT = The expected swap usage.
MEM – SWR/s (J) = If larger than 0 host is actively reading from swap(vswp).
MEM – SWW/s (J) = If larger than 0 host is actively writing to swap(vswp).

So which metrics do really matter when your customer complains about degradation of performance?

First metric to check:
SWR/s (J) = If larger than zero the ESX host is actively reading from swap(vswp).

Associated to that metric I would recommend looking at the following metric:
%SWPWT = The percentage of time the world is waiting for the ESX VMKernel swapping memory.

So what about all those other metrics? Why don’t they really matter?
Take “Current Swap”, as long as it is not being “read” it might just be one of those pages sporadically used which is just sitting there doing nothing. Will it hurt performance? Maybe, but currently as long as it is not being read… no it will most likely not hurt. Even writing to swap does not necessarily hurt performance, it might though. Those should just be used as indicators that the system is severely overcommitted and that performance might be degraded in the future when pages are being read!

Re: VMware vSphere 4 default installation settings (gabesvirtualworld)

· ·

In response to Gabes article on default installation settings there are some things I personally almost always do different and I wanted to point them out. Consider them my recommendations / best practices and not necessary VMware’s. I’ve added two (*) and have a different opinion on some of Gabe’s best practices (-)

COS Memory:

  • Although COS memory is “dynamic” I still always increase it to the full 800. The overhead of this in most of the servers(usually always 48GB+) is tiny. (-)

Host Configuration:

  • Hostnames in lowercase characters; to avoid any HA issues. (*)
  • I never change the name of the Service console portgroup, people are used to this name changing it leads to confusion in most cases and it is a critical part of your host. (-)
  • Avoid using agents within the Service Console. (*)

vSwitch settings:

  • Mac address changes: Reject (-)
    A best practice recommended by VMware PSO to ensure that when someone changes a MAC within the OS all inbound packets are dropped.
  • Forged Transmit: Reject (-)
    Setting Forged Transmits to reject ensures that the originator of the packet is validated. Any outbound frame with a MAC address that is different from the one currently set on the adapter will be dropped. Again a best practice recommended by VMware PSO.