Yellow Bricks

by Duncan Epping

memory tiering

Memory Tiering Considerations and frequently asked questions!

· · 2 Comments

Over the last couple of months I’ve been presenting on Memory Tiering at various events, and the same questions and comments keep coming up. Hence, I figured it would make sense to create a considerations and frequently asked questions blog. This will hopefully make it easier for people to find answers to the questions they have. If I missed out on something, feel free to leave a comment.

Now, before I write down any commonly asked questions, and things people should know, I want to point you to this fantastic Memory Tiering Resources page. Dave Morera has written a series of great articles, that will actually answer most questions people may have, so make sure to head over there as well.

  • Is Memory Tiering not just swapping to NVMe?
    No, with swapping random pages are moved to disk to avoid memory filling up. With memory tiering the memory of each VM is scanned in batches, and then classified (when was the page last accessed) during each pass. Memory pages are moved only to NVMe when they are classified as cold (or very cold).
  • Which NVMe device should I use, and how do I know which one is supported?
    The requirements are documented here. You can look at the vSAN SSD Compatibility Guide here to find the supported devices.
  • What is the preferred method of configuring Memory Tiering?
    Starting with 9.1 memory tiering can be fully configured using the “esxcli memtier” namespace, or through Configuration Profiles. There’s no preferred option.
  • When does Memory Tiering become active?
    When Memory Consumption is roughly 80%, the algorythm will start considering to move memory pages from DRAM to NVMe.
  • Can I change the percentage at which memory tiering starts moving pages from 80% to 90% or any other value above or below that 80%?
    No, this is not supported. It is a fixed value!
  • What about large pages?
    When memory tiering is enabled, large pages are disabled on each VM. Memory tiering works on a 4KB granularity, you can imagine that it is virtually impossible to find large pages which are completely cold, as a result it doesn’t make sense to consider those really for Memory Tiering.
  • But if I have a stretched cluster configuration typically I don’t run at above 50% consumed to allow for a failure of a datacenter?
    Currently memory tiering does not take that scenario into consideration.
  • When is an environment too active for Memory Tiering to be useful?
    In general VMware recommends “active memory” to be lower than 50%
  • Are there other recommendations when it comes to resource management before enabling Memory Tiering?
    Yes, you should verify whether memory consumption is high enough for memory tiering to become active. And also consider why you are enabling memory tiering. If CPU usage is high and memory usage is high, what are you achieving by enabling memory tiering? Most likely it will not improve the consolidation ratio for instance. But if CPU usage is low, and memory consumption is high, you could use it to increase CPU usage.
  • Will Memory Tiering also move memory pages of the kernel if they are cold?
    No, Memory Tiering only considers VM memory pages, kernel pages (etc) are not classified and moved.
  • Does Memory Tiering support NVMe devices in a RAID-1 configuration?
    Yes, RAID-1 is supported both via a hardware controller and starting with 9.1 in software as well.
  • If I use Memory Tiering in a RAID-1 configuration, are reads distributed?
    Yes, reads are done in a round robin fashion across both NVMe devices in the RAID pair.
  • What happens if an NVMe device fails which is used for Memory Tiering?
    If you have RAID-1 configured (software or hardware) then the VM will remain running as the memory pages will be accessible on the remaining device. If you don’t have RAID-1 configuration, it is like a DRAM DIMM failure, and the VM will be restarted (HA event) when a page is being accessed on the failed device.
  • What happens if I set a memory reservation while Memory Tiering is enabled, is the reservation always backed by DRAM?
    No, the memory reservation can be backed by DRAM and NVMe, the host sees the combined total is one large pool of physical memory
  • Can I power-on VMs which have FT enabled, or passthrough devices configured on a host with memory tiering?
    Yes, starting with 9.1 this is possible. However, keep in mind that memory tiering will be explicitly disabled for those VMs!
  • Is the Marvell Structera X 2404 controller supported with vSphere or Memory Tiering?
    The Marvell Structera X 2404 controller enables you to present DDR-4 memory via a PCIe bus using CXL as a large memory pool to the host. Unfortunately this has not been officially tested with vSphere, and as such also not with Memory Tiering.
  • Are there any performance numbers that can be shared?
    For vSphere 9.0 extensive performance tests have been conducted with memory tiering, these have been shared in this whitepaper by the performance team that can be found here. With 9.1 various benchmarks have shown a performance improvement over 9.0 of 16% while reducing CPU overhead with 12% when using Memory Tiering, this is documented in this blog here.
  • Should I set my memory tiering ratio to 1:1, 2:1, 4:1, 1:2 etc?
    Your memory tiering ratio should be based on the workload running on the host/cluster. Some workloads are more sensitive to latency then others, you need to take that into consideration. Also, some workloads have a higher active memory percentage then others, this also impacts how much memory there will be tiered at most.
  • What is the impact of the ratio on my NVMe device?
    The ratio determines the allocated size on the NVMe device for tiering. If you have a ratio of 1:1, and you have 512GB of DRAM and a 2TB NVMe device, then the amount of NVMe used for memory tiering will be 512GB.
  • What happens with the excess capacity on an NVMe device in the previous example?
    This capacity does not go unused, the NVMe device will use the cells to cycle through them for endurance.
  • Is there a maximum size for the partition on the NVMe device that is used for memory tiering?
    Yes, the maximum size is 4TB at the point of writing. (version 9.1)
  • Can I use Memory Tiering in combination with Nested Virtualization? Yes, you can set the advanced setting sched.mem.enableNestedTiering to TRUE, as described here. Do note, this should be used for testing purposes only.
  • Can I disable Memory Tiering for a specific VM?
    Yes, you can. Simply add sched.mem.enableTiering to the advanced setting of the VM and set it to FALSE.
  • Can I enable encryption on a per VM basis with Memory Tiering?
    Yes, you can enable it for the full host, or encrypt per VM via the advanced configuration sched.mem.EncryptTierNVMe=TRUE.
  • What happens if you vMotion a VM which has memory tiered?
    When you migrate / vMotion a VM which has memory pages tiered then the vMotion process functions as expected. The only real difference is that for pages which are on NVMe, the pages will need to be read from NVMe in to DRAM and then transferred over the network. As a result the vMotion process will be slower for VMs which have tiered memory pages compared to VMs that only have pages in DRAM. When the VM is moved from source to destination, in the destination all memory paged will be in DRAM and may over time move to NVMe if memory tiering is enabled on the host.

 

Enabling Memory Tiering in VCF/vSphere 9.1 through the CLI!

· · Leave a Comment

I saw a question internally this week about enabling Memory Tiering through the CLI with 9.1 and why that didn’t work as you would expect. I wrote an article about the esxcli commands a while back, so I went through them and then I realized indeed that the article was outdated. I participated in a few early access programs and one of the things we had to do was enable memory tiering using the new esxcli namespace. Where in the past there were  various advanced settings you had to configure, and various commands to run, they now made things much easier. Just some examples below.

The new name space:

esxcli memtier

Enable memory tiering using a particular device:

esxcli memtier enable -d <device_name>

Enable memory tiering using a particular device and with a 400% ratio percentage, this value can be between 1 and 400:

esxcli memtier enable -d <device_name> -r 400%

Enable encryption after memory tiering has been configured:

esxcli memtier config set --encryption true

You can find the full documentation here and here.

Playing around with Memory Tiering, are my memory pages tiered?

· · 2 Comments

There was a question on VMTN about Memory Tiering performance, and how you can check if pages were tiered. I haven’t played around with Memory Tiering too much, so I noted down for myself what I needed to do on every host in order to enable it. Note, if the command contains a path and you want to do this in your own environment you need to change the path and device name accordingly. The question was if memory pages were tiered or not, so I dug up the command that allows you to check this on a per host level. It is at the bottom of this article for those who just want to skip to that part.

Now, before I forget, probably worth mentioning as this is something many people don’t seem to understand, memory tiering only tiers cold memory pages. Active pages are not being moved to NVMe, on top of that, it only tiers memory when there’s memory pressure! So if you don’t see any tiering, it could simply be that you are not under any memory capacity pressure. (Why move pages to a lower tier when there’s no need?)

Please note, the below commands apply for all versions below vSphere/VCF 9.1! For VCF/vSphere 9.1 please look at this blog post!

List all storage devices via the CLI:

esxcli storage core device list

Create memory tiering partition on an NVMe device:

esxcli system tierdevice create -d=/vmfs/devices/disks/eui.1ea506b32a7f4454000c296a4884dc68

Enable Memory Tiering on a host level, note this requires a reboot:

esxcli system settings kernel set -s MemoryTiering -v TRUE

How is Memory Tiering configured in terms of DRAM to NVMe ratio? A 4:1 DRAM to NVMe ratio would be 25%, 1:1 would be 100%. So if you have it set at 4:1, with 512GB of DRAM you would only use 128GB of the NVMe at most, regardless of the size of the device.

esxcli system settings advanced list -o /Mem/TierNvmePct

Is memory tiered or not? Find out all about it via memstats!

memstats -r vmtier-stats -u mb

Want to show a select number of metrics?

memstats -r vmtier-stats -u mb -s name:memSize:active:tier1Target:tier1Consumed:tier1ConsumedPeak:comnsumed

So what would the outcome look like when there is memory tiering happening? I removed a bunch of the metrics, just to keep it readable, “tier1” is the NVMe device, and as you can see each VM has several MBs worth of memory pages on NVMe right now.

 VIRTUAL MACHINE MEMORY TIER STATS: Wed Dec 17 15:29:43 2025
 -----------------------------------------------
   Start Group ID   : 0
   No. of levels    : 12
   Unit             : MB
   Selected columns : name:memSize:tier1Consumed

----------------------------------------
           name    memSize tier1Consumed
----------------------------------------
      vm.533611       4096            12
      vm.533612       4096            34
      vm.533613       4096            24
      vm.533614       4096            11
      vm.533615       4096            25
----------------------------------------
          Total      20480           106
----------------------------------------

#103 – The performance impact of Memory Tiering featuring Qasim Ali and Todd Muirhead

· · Leave a Comment

The last few months, I’ve had many discussions about Memory Tiering. When I saw a brand new performance white paper being released, I knew it was time to invite two of the authors to the podcast. Qasim Ali and Todd Muirhead go over the ins and outs of Memory Tiering, they discuss the basics, but also explain in-depth what the potential performance impact is when enabling this feature in your environment. You can listen on Apple Podcasts, Spotify, the embedded player below, or any podcast app of your choice!

If you’d like to know more, visit the following links!

Memory Tiering… Say what?!

· ·

Recently I presented a keynote at the Belgium VMUG, the topic was Innovation at VMware by Broadcom, but I guess I should say Innovation at Broadcom to be more accurate. During the keynote I briefly went over the process and the various types of innovation and what this can lead to. During the session, I discussed three projects, namely vSAN ESA, the Distributed Services Engine, and something which is being worked on called: Memory Tiering.

Memory Tiering is a very interesting concept that was first publicly discussed at Explore (or VMworld I guess it was still. called) a few years ago as a potential future feature. You may ask yourself why anyone would want to tier memory, as the impact from a performance stance can be significant. There are various reasons to do so, one of them being the cost of memory. Another problem the industry is facing is the fact that memory capacity (and performance) has not grown at the same rate as CPU capacity, which has resulted in many environments being memory-bound, differently said the imbalance between CPU and memory has increased substantially. That’s why VMware started Project Capitola.

When Project Capitola was discussed most of the focus was on Intel Optane, and most of us know what happened to that. I guess some assumed that that would also result in Project Capitola, or memory tiering and memory pooling technology, being scrapped. This is most definitely not the case, VMware has gone full steam ahead and has been discussing the progress in public, although you need to know where to look. If you listen to that session it is clear that there are various efforts, that would allow customers to tier memory in various ways, one of them being of course the various CXL based solutions that are coming to market now/soon.

One of which is memory tiering via a CXL accelerator card, basically an FPGA that has the sole purpose of increasing memory capacity, offload memory tiering and accelerating certain functionality where memory is crucial like for instance vMotion. As mentioned in the SNIA session, using an accelerator card can lead to a 30% reduction in migration times. An accelerator card like this will also open up other opportunities, like pooling memory for instance, which is something customers have been asking for since we created the concept of a cluster. Being able to share compute resources across hosts. Just imagine, your VM can use memory capacity available on another host without having to move the VM. Yes, before anyone comments on this, I do realize that this will have a significant performance impact potentially.

That is of course where the VMware logic comes into play. At VMworld in 2021 when Project Capitola was presented, the team also shared the performance results of recent tests, and it showed that the performance degradation was around 10% when 50% of DRAM was used and 50% of Optane memory. I was watching the SNIA session, and this demo shows the true power of VMware vSphere, memory tiering, and acceleration (Project Peaberry as it is called). On average the performance degradation was around 10%, yet roughly 40% of virtual memory was accessed via the Peaberry accelerator. Do note that the tiering is completely transparent to the application, this works for all different types of workloads out there. The crucial part here to understand is that because the hypervisor is already responsible for memory management, it knows which pages are hot and which pages are cold, that also means it can determine which pages it can move to a different tier while maintaining performance.

Anyway, I cannot reveal too much about what may, or may not, be coming in the future. What I can promise though is that I will make sure to write a blog as soon as I am allowed to talk about more details publicly, and I will probably also record a podcast with the product manager(s) when the time is there, so stay tuned!