Yellow Bricks

by Duncan Epping

numa

vNUMA and vMotion

· ·

I was listening to some VMworld talks during the weekend and something caught my attention which I hadn’t realized before. The talk I was listening to was VSP2122″VMware vMotion in vSphere 5.0, Architecture and Performance”. Now this probably doesn’t apply to most of the people reading this so let me set the scenario first:

  • Different hosts from a CPU/Memory perspective in a single cluster (different NUMA topology)
  • VMs with more than 8 vCPUs

Now the thing is that the vNUMA topology is set for a given VM during the power-on. This is based on the NUMA topology of the physical host that has received the power-on request. When you move a VM to a host which has a different NUMA topology then it could result in reduced performance. This is also described in the Performance Best Practices whitepaper for vSphere 5.0. A nice example of how you can benefit from vNUMA is explained in the recently released academic paper “Performance Evaluation of HPC Benchmarks on VMware’s ESXi Server“.

I’ve never been a huge fan of mixed clusters due to complications it adds around resource management and availability, but this is definitely another argument to try to avoid it where and when possible.

Disabling TPS hurting performance?

· ·

On the internal mailinglist there was a discussion today around how disabling TPS (Transparent Page Sharing) could negativitely impact performance. It is something I hadn’t thought about yet but when you do think about it it actually does make sense and is definitely something to keep in mind.

Most new servers have some sort of NUMA architecture today. As hopefully all of you know TPS does not cross a NUMA node boundary. This basically means that pages will not be shared between NUMA nodes. Another thing that Frank Denneman already described in his article here is that when memory pages are allocated remotely there is a memory penalty associated with it. (Did you know there is an “esxtop” metric, N%L,which shows the percentage of remote pages?) These pages are accessed across an interconnect bus which is always slower than so called local memory.

Now you might ask what is the link between NUMA, TPS and degraded performance? Think about it for a second… TPS decreases the amount of physical pages needed. If TPS is disabled there is no sharing and chances of going across NUMA nodes are increased and as stated this will definitely impact performance. Funny how disabling a mechanism(TPS) which is often associated with “CPU overhead” can have a negative impact on memory latency.