I was just reading a discussion on the VMTN community on CPU affinity. The general opinion of the Experts is “Don’t use CPU affinity”. I fully agree with them, ESX is more than capable to handle the scheduling on it’s own with just a limited overhead. And like Ken Cline also stresses it could harm performance because of NUMA load balancing for instance.
Something that’s often overlooked though when one does CPU affinity is that people tend to give the VM vCPUs a 1:1 relationship with host cores. In other words a VM with two vCPUs will be pinned down to two cores on the host.
This does make sense doesn’t it? No it actually doesn’t. There’s more to a VM than just it’s vCPUs. I would like to refer to page 132 of the Resource Management Guide, aka HA-DRS Bible. In short, besides the vCPUs there are several VM associated threads that need to be scheduled as well. When affinity is set these threads, or worlds as VMware calls them, will be scheduled on the assigned cores. You can imagine that when you use the vCenter client to manage the client these threads(Video / Keyboard / Mouse / CD-Rom etc) will need to be scheduled on the same set of cores as the vCPUs need to be scheduled on… If you have a two vCPU VM and want to use CPU affinity pin it down to at least three cores! Before you start assigning cores to your VM also read the bulletpoints on page 133 why you shouldn’t.
The CPU affinity setting for a virtual machine applies not only to all of the virtual CPUs associated with the virtual machine, but also to all other threads (also known as “worlds”) associated with the virtual machine. Such virtual machine threads perform processing required for emulating mouse, keyboard, screen, CD‐ROM and miscellaneous legacy devices.
In some cases, such as display‐intensive workloads, significant communication might occur between the virtual CPUs and these other virtual machine threads. Performance might degrade if the virtual machineʹs affinity setting prevents these additional threads from being scheduled concurrently with the virtual machineʹs virtual CPUs (for example, a uniprocessor virtual machine with affinity to a single CPU, or a two‐way SMP virtual machine with affinity to only two CPUs).
For the best performance, when manual affinity settings are used, VMware recommends that you include at least one additional physical CPU in the affinity setting in order to allow at least one of the virtual machineʹs threads to be scheduled at
the same time as its virtual CPUs (for example, a uniprocessor virtual machine with affinity to at least two CPUs or a two‐way SMP virtual machine with affinity to at least three CPUs).
Agreed on general rule. The one environment I have seen affinity *required* for a supported configuration is Cisco’s Unity deployment on VI3. But they missed the point about +1 core as well.
http://www.cisco.com/en/US/docs/voice_ip_comm/unity/virtualization_design/guide/cuvirtualdg010.html#wp61553
All I can say is that there are times when however unpalatable it is, CPU Affinity can be used to overcome issues and problems. For example…say you have a VM that is clearly CPU bound and is right on the edge of being able to run on your current host architecture…using CPU Affinity to keep a VM from executing on Core 0, where the SC currently is ALWAYS scheduled…could eke out a few small percents of performance gains for you. But yes…I also agree that Affinity should be avoided whenever possible.
CPU Affinity doesnt work together with vMotion and Storage vMotion. Its allso not possible to activate HA if one of the VMs have set a CPU affinity.
In a HA or DRS cluster you cant set CPU Affinity because the boxes are grey out.
Regards
Joerg
I wrote the Cisco Unity document referenced and did all the testing behind the solution. For 18 months it was my baby.
I arrived at the CPU Affinity requirement because it was the only way to maintain voice quality with other applications running side by side. This severely limited the solution but it was the reality of the beast.
I’m not questioning the technical merit behind the +1 but suggest that it’s not a hard requirement. As VMware says, +1 gives you the best performance but that doesn’t mean your app can’t get by with out it. During the Unity on VMware load testing I monitored voice quality closely and didn’t experience hiccups (scheduling issues) when accessing the console without using +1.
I recommend people test their apps with end user perception in mind. Start with the least restrictive CPU protection strategy possible. Is end user perception acceptable? If not, take it up a notch and repeat. If you do have to use CPU Affinity, see if you can get away without the +1.
Keith, thanks for the comment but keep in mind that you were working on a unique situation where the host probably wasn’t overcommitted on CPU. When it is you will start noticing that scheduling isn’t as easy as expected especially not when you have a VM which has a lot of interaction going on and is actually using those helper worlds.
I agree that it is something that needs to be tested, but i would rather have an extra than running low with these kind of apps.
Excellent breakdown of the perils of CPU Affinity.
I wasn’t planning on using it but the threads/worlds were an aspect I hadn’t considered, all the more reason to avoid!
Just doing a bit of reading on cache coherency and data locality under SQL 2008, and a strong theme is the cost of a having to go back to system memory to access data which cached on another CPU (be it another socket or core – cores share L2, but rarely L1 cache). I totally understand the potential issues with NUMA (you’d mess things up nicely if you restricted a VM to using CPUs in disparate NUMA groups), and understand the other restrictions within VMware itself (no HA or vMotion), could cache coherency and data locality be a reason to consider CPU affinity, or is that really too narrow a use case to consider except under extreme circumstances?