ESX

I was just reading a discussion on the VMTN community on CPU affinity. The general opinion of the Experts is “Don’t use CPU affinity”. I fully agree with them, ESX is more than capable to handle the scheduling on it’s own with just a limited overhead. And like Ken Cline also stresses it could harm performance because of NUMA load balancing for instance.

Something that’s often overlooked though when one does CPU affinity is that people tend to give the VM vCPUs a 1:1 relationship with host cores. In other words a VM with two vCPUs will be pinned down to two cores on the host.

This does make sense doesn’t it? No it actually doesn’t. There’s more to a VM than just it’s vCPUs. I would like to refer to page 132 of the Resource Management Guide, aka HA-DRS Bible. In short, besides the vCPUs there are several VM associated threads that need to be scheduled as well. When affinity is set these threads, or worlds as VMware calls them, will be scheduled on the assigned cores. You can imagine that when you use the vCenter client to manage the client these threads(Video / Keyboard / Mouse / CD-Rom etc) will need to be scheduled on the same set of cores as the vCPUs need to be scheduled on… If you have a two vCPU VM and want to use CPU affinity pin it down to at least three cores! Before you start assigning cores to your VM also read the bulletpoints on page 133 why you shouldn’t.

The CPU affinity setting for a virtual machine applies not only to all of the virtual CPUs associated with the virtual machine, but also to all other threads (also known as “worlds”) associated with the virtual machine. Such virtual machine threads perform processing required for emulating mouse, keyboard, screen, CD‐ROM and miscellaneous legacy devices.

In some cases, such as display‐intensive workloads, significant communication might occur between the virtual CPUs and these other virtual machine threads. Performance might degrade if the virtual machineʹs affinity setting prevents these additional threads from being scheduled concurrently with the virtual machineʹs virtual CPUs (for example, a uniprocessor virtual machine with affinity to a single CPU, or a two‐way SMP virtual machine with affinity to only two CPUs).

For the best performance, when manual affinity settings are used, VMware recommends that you include at least one additional physical CPU in the affinity setting in order to allow at least one of the virtual machineʹs threads to be scheduled at
the same time as its virtual CPUs (for example, a uniprocessor virtual machine with affinity to at least two CPUs or a two‐way SMP virtual machine with affinity to at least three CPUs).