Yellow Bricks

Over the last couple of months I’ve been presenting on Memory Tiering at various events, and the same questions and comments keep coming up. Hence, I figured it would make sense to create a considerations and frequently asked questions blog. This will hopefully make it easier for people to find answers to the questions they have. If I missed out on something, feel free to leave a comment.

Now, before I write down any commonly asked questions, and things people should know, I want to point you to this fantastic Memory Tiering Resources page. Dave Morera has written a series of great articles, that will actually answer most questions people may have, so make sure to head over there as well.

• Is Memory Tiering not just swapping to NVMe?
  No, with swapping random pages are moved to disk to avoid memory filling up. With memory tiering the memory of each VM is scanned in batches, and then classified by the hypervisor (when was the page last accessed) during each pass. Memory pages are moved only to NVMe when they are classified as cold (or very cold).
• Which NVMe device should I use, and how do I know which one is supported?
  The requirements are documented here. You can look at the vSAN SSD Compatibility Guide here to find the supported devices.
• What is the preferred method of configuring Memory Tiering?
  Starting with 9.1 memory tiering can be fully configured using the “esxcli memtier” namespace, or through Configuration Profiles. There’s no preferred option.
• When does Memory Tiering become active?
  When Memory Consumption is roughly 80%, the algorythm will start considering to move cold memory pages from DRAM to NVMe.
• How often does memory tiering scan all memory pages?
  This is a constant process.
• Can I change the percentage at which memory tiering starts moving pages from 80% to 90% or any other value above or below that 80%?
  No, this is not supported. It is a fixed value!
• What about large pages?
  When memory tiering is enabled, large pages are disabled on each VM. Memory tiering works on a 4KB granularity, you can imagine that it is virtually impossible to find large pages which are completely cold, as a result it doesn’t make sense to consider those really for Memory Tiering.
• But if I have a stretched cluster configuration typically I don’t run at above 50% consumed to allow for a failure of a datacenter?
  Currently memory tiering does not take that scenario into consideration.
• When is an environment too active for Memory Tiering to be useful?
  In general VMware recommends “active memory” to be lower than 50%
• Are there other recommendations when it comes to resource management before enabling Memory Tiering?
  Yes, you should verify whether memory consumption is high enough for memory tiering to become active. And also consider why you are enabling memory tiering. If CPU usage is high and memory usage is high, what are you achieving by enabling memory tiering? Most likely it will not improve the consolidation ratio for instance. But if CPU usage is low, and memory consumption is high, you could use it to increase CPU usage.
• Will Memory Tiering also move memory pages of the kernel if they are cold?
  No, Memory Tiering only considers VM memory pages, kernel pages (etc) are not classified and moved.
• Does Memory Tiering support NVMe devices in a RAID-1 configuration?
  Yes, RAID-1 is supported both via a hardware controller and starting with 9.1 in software as well.
• If I use Memory Tiering in a RAID-1 configuration, are reads distributed?
  Yes, reads are done in a round robin fashion across both NVMe devices in the RAID pair.
• What happens if an NVMe device fails which is used for Memory Tiering?
  If you have RAID-1 configured (software or hardware) then the VM will remain running as the memory pages will be accessible on the remaining device. If you don’t have RAID-1 configuration, it is like a DRAM DIMM failure, and the VM will be restarted (HA event) when a page is being accessed on the failed device.
• What happens if I set a memory reservation while Memory Tiering is enabled, is the reservation always backed by DRAM?
  No, the memory reservation can be backed by DRAM and NVMe, the host sees the combined total is one large pool of physical memory
• Can I power-on VMs which have FT enabled, or passthrough devices configured on a host with memory tiering?
  Yes, starting with 9.1 this is possible. However, keep in mind that memory tiering will be disabled for those VMs!
• But what about the memory reservation then for VMs with passthrough devices then?
  For those VMs, as Memory Tiering is disabled implicitly, the memory reservation will be backed by DRAM only.
• Is the Marvell Structera X 2404 controller supported with vSphere or Memory Tiering?
  The Marvell Structera X 2404 controller enables you to present DDR-4 memory via a PCIe bus using CXL as a large memory pool to the host. Unfortunately this has not been officially tested with vSphere, and as such also not with Memory Tiering.
• Are there any performance numbers that can be shared?
  For vSphere 9.0 extensive performance tests have been conducted with memory tiering, these have been shared in this whitepaper by the performance team that can be found here. With 9.1 various benchmarks have shown a performance improvement over 9.0 of 16% while reducing CPU overhead with 12% when using Memory Tiering, this is documented in this blog here.
• But what is the overhead of memory tiering on CPU?
  The documentation states the following: Software CPU overhead depends on the host memory, and you can see a small usage of about an eighth of a core for Memory Tiering for smaller-sized hosts.
• Should I set my memory tiering ratio to 1:1, 2:1, 4:1, 1:2 etc?
  Your memory tiering ratio should be based on the workload running on the host/cluster. Some workloads are more sensitive to latency then others, you need to take that into consideration. Also, some workloads have a higher active memory percentage then others, this also impacts how much memory there will be tiered at most.
• What is the impact of the ratio on my NVMe device?
  The ratio determines the allocated size on the NVMe device for tiering. If you have a ratio of 1:1, and you have 512GB of DRAM and a 2TB NVMe device, then the amount of NVMe used for memory tiering will be 512GB.
• What happens with the excess capacity on an NVMe device in the previous example?
  This capacity does not go unused, the NVMe device will use the cells to cycle through them for endurance.
• Is there a maximum size for the partition on the NVMe device that is used for memory tiering?
  Yes, the maximum size is 4TB at the point of writing. (version 9.1)
• Can I use Memory Tiering in combination with Nested Virtualization? Yes, you can set the advanced setting sched.mem.enableNestedTiering to TRUE, as described here. Do note, this should be used for testing purposes only.
• Can I disable Memory Tiering for a specific VM?
  Yes, you can. Simply add sched.mem.enableTiering to the advanced setting of the VM and set it to FALSE.
• Can I enable encryption on a per VM basis with Memory Tiering?
  Yes, you can enable it for the full host, or encrypt per VM via the advanced configuration sched.mem.EncryptTierNVMe=TRUE.
• Are Self Encrypting Devices (SED) supported?
  Memory Tiering does not support the use of SED’s unfortunately, this is planned for a future release.
• What happens if you vMotion a VM which has memory tiered?
  When you migrate / vMotion a VM which has memory pages tiered then the vMotion process functions as expected. The only real difference is that for pages which are on NVMe, the pages will need to be read from NVMe in to DRAM and then transferred over the network. As a result the vMotion process will be slower for VMs which have tiered memory pages compared to VMs that only have pages in DRAM. When the VM is moved from source to destination, in the destination all memory paged will be in DRAM and may over time move to NVMe if memory tiering is enabled on the host.