ESX

I was just reading Scott Drummonds article on Memory Compression. Scott explains where Memory Compression comes in to play. I guess the part I want to reply on is the following:

VMware’s long-term prioritization for managing the most aggressively over-committed memory looks like this:

1. Do not swap if possible.  We will continue to leverage transparent page sharing and ballooning to make swapping a last resort.
2. Use ODMC to a predefined cache to decrease memory utilization.*
3. Swap to persistent memory (SSD) installed locally in the server.**
4. Swap to the array, which may benefit from installed SSDs.

(*) Demonstrated in the lab and coming in a future product.
(**) Part of our vision and not yet demonstrated.

I just love it when we give insights in upcoming features but I am not sure I agree with the prioritization. I think there are several things that one needs to keep in mind. In other words there’s a cost associated to these decisions / features and your design needs to adjusted to these associated effects.

1. TPS -> Although TPS is an amazing way of reducing the memory footprint you will need to figure out what the ratio of deduplication is. Especially when you are using Nehalem processors there’s a serious decrease. The reasons for the decrease of TPS effectiveness are the following:
   • NUMA – By default there is no inter node transparent page sharing (read Frank’s article for more info on this topic)
   • Large Pages – By default TPS does not share large(2MB) pages. TPS only shares small(4KB) pages. It will break large pages down in small pages when memory is scarce but it is definitely something you need to be aware off. (for more info read my article on this topic.
2. Use ODMC -> I haven’t tested with ODMC yet and I don’t know what the associated cost is at the moment.
3. Swap on local SSD -> Swap on local SSD will most definitely improve the speed when swapping occurs. However as Frank already described in his article there is an associated cost:
   • Disk space – You will need to make sure you will have enough disk space available to power on VMs or migrate VMs as these swap files will be created at power on or at migration.
   • Defaults – By default .vswp files are stored in the same folder as the .vmx. Changing this needs to be documented and taken into account during upgrades and design changes.
4. Swap to array (SSD) -> This is the option that most customers use for the simple reason that it doesn’t require a local SSD disk. There are no changes needed to enable it and it’s easier to increase a SAN volume than it is to increase a local disk when needed. The associated costs however are:
   • Costs – Shared storage is relatively expensive compared to local disks
   • Defaults – If .vswp files need to be SSD based you will need to separate the .vswp from the rest of the VMs and created dedicated shared SSD volumes.

I fully agree with Scott that it’s an exciting feature and I can’t wait for it to be available. Keep in mind though that there is a trade off for every decision you make and that the result of a decision might not always end up as you expected it would. Even though Scott’s list makes totally sense there is more than  meets the eye.