Yellow Bricks

by Duncan Epping

Server

Host placement for the secondary FT VM with Fault Domains?

· · Leave a Comment

On Reddit there was a question about how to specify on which group of hosts the secondary VM of an FT-enabled VM would land on. Now this is a problem that has been around for a while, and for the longest time we actually did not have a solution for it, but now we do. I did a quick search, and somehow I couldn’t find an article where I describe how this works, so I figured I would write one. Having said that, I would not be surprised if there’s already and article out there, but somehow didn’t pop up.

When it comes to vSphere Fault Tolerance, when you enable it on a VM, what you will see happening is that the VM you enabled it on will get “(primary)” behind the name. This VM is, as you would expect, the primary and is paired with a “secondary”. This “secondary” VM runs in lockstep with the primary, so that it can take over instantly if the primary would fail for any reason. Now, that secondary VM is kind of a strange VM. Although you can find the VM in your inventory on a host level, you cannot find it on a cluster level or in the VM view. It is like a shadow copy, it is there, but yet it isn’t really.

As a result, making VM-Host rules for it is rather difficult. Although you can do it, this shadow / secondary VM can also be reinstantiated and as a result that VM-Host rule would unfortunately break. Why? Well, the newly instantiated VM is literally that… a newly instantiated VM, so the old rules do not apply. So although the VM-Host rules with tags can work through the Compute Policy option, it not ideal. Probably should also mention, traditional VM-Host groups and rules do not even work with the secondary VM, as you simply cannot select the secondary VM to begin with.

So what does work? Well, there’s a new option called Enable Metro Cluster Fault Tolerance. What this does is that when you select a Host Group for the primary, the secondary VM automatically ends up on a host outside of the selected group. The Host Group can simply be defined via the VM-Host group option under your cluster’s Configure tab as shown in the screenshots below.

Hopefully this helps those who are planning on leveraging FT and want to ensure the primary and secondary are placed on different fault domains.

Host placement for the secondary FT VM with Fault Domains?

 

Memory Tiering aware vSphere HA Admission Control in VCF/vSphere 9.1.1!

· · Leave a Comment

To be fair, I have not had many customers complain about the fact that vSphere HA Admission Control was not Memory Tiering aware, but if you had concerns then it is good to know that this problem was resolved with the 9.1.1 release of VCF/vSphere. For those who never gave it any thought even, what was the problem?

When Memory Tiering is enabled, vSphere and vSphere HA considers the total amount of DRAM and the amount of NVMe of the Memory Tier as the total amount of available memory. So if you have 512GB of DRAM, and 512GB of NVMe for Memory Tiering, it would be considered as a single pool of memory of 1TB. Of course, this is the whole idea behind Memory Tiering, you have a hot and a cold tier and memory pages move up and down based on access patterns.

However, this only applies to memory pages of a VM (so for instance not of vSAN). On top of that, it only applies to memory pages which are not reserved and pinned to DRAM. On top of that, the memory overhead of a VM is also never considered for the NVMe tier. You can imagine that in certain scenarios, you could end up in the situation where vSphere HA expects there to be memory available, and even though there’s capacity in the memory pool, it is only in NVMe and not in DRAM. If that happens, it becomes impossible to restart a VM, as for instance the memory overhead always needs to be backed by DRAM.

As a result, starting with VCF/vSphere 9.1.1, the vSphere HA team has included logic for HA Admission Control to differentiate between DRAM and NVMe, and also between a tiered VM and a non-tiered VM. This by the way, only applies to the “Percentage based” admission control policy, and the “Failover Host” admission control policy. The slot policy does not differentiate between these!

Now when it comes to Admission Control and Memory Tiering, Admission Control will actually take the total amount of memory in to account (DRAM+NVMe tier) but it will also look at the capacity of DRAM only and consider that. When you set for instance the percentage to 25%, it will do the math for both DRAM as well as DRAM+NVMe, and new workloads will not be accepted unless they would pass both admission control calculations. This to prevent a situation where memory pages can be backed by NVMe and DRAM but for instance a reservation of the memory overhead cannot be guaranteerd by DRAM alone, as that would result in a failed restart.

There documentation of vSphere HA has been updated to reflect the latest changes, if you use Memory Tiering and have Admission Control enabled I would recommend looking at the documentation for the various examples of how it works. I would also highly recommend customers who use Memory Tiering and HA Admission Control to verify which policy has been selected, as that will also have an impact as explained above.

PS: the memory tiering aware HA Admission Control only comes in to play when DRS is enabled, and vSphere HA can call DRS for placement recommendations. It does not work when vCenter is unavailable and the SPE is used.

vSphere HA setting Performance degradation VMs tolerate

· · Leave a Comment

There was a question this week internally and I really had to start digging, as I have not looked at this in a loooong time. What does “Performance degradation VMs tolerate” do? And does this feature require admission control to be enabled or not?

vSphere HA setting Performance degradation VMs tolerate

I had to test this, as I barely ever play around with the HA settings these days. But, let’s first describe what this feature is for. I think the UI explains it fairly decently, but here’s my explanation from the vSphere Clustering Deep Dive:

This feature allows you to specify the performance degradation you are willing to incur if a failure happens. It is set to 100% by default, but it is our recommendation to consider changed the value. You can for instance change this to 25% or 50.

Now, the requirement for this feature to work is to have DRS enables, but Admission Control does not need to be enabled! A lot of people are under the impression that it requires Admission Control in order to take “an X number of failures” into account, but it does not. It actually does not use what is specified for Admission Control. It takes a single failure into account when it comes to this feature, and then uses DRS to do the calculations if powered on VMs will get the same amount of resources allocated after a failure. If the answer is no, or performance degradation is higher than the percentage specified, a warning is triggered. You will still be able to power on new VMs, but the warning will not go away unless the resource usage changes, or you add more resources to the cluster.

Playing around with Memory Tiering, are my memory pages tiered?

· · 2 Comments

There was a question on VMTN about Memory Tiering performance, and how you can check if pages were tiered. I haven’t played around with Memory Tiering too much, so I noted down for myself what I needed to do on every host in order to enable it. Note, if the command contains a path and you want to do this in your own environment you need to change the path and device name accordingly. The question was if memory pages were tiered or not, so I dug up the command that allows you to check this on a per host level. It is at the bottom of this article for those who just want to skip to that part.

Now, before I forget, probably worth mentioning as this is something many people don’t seem to understand, memory tiering only tiers cold memory pages. Active pages are not being moved to NVMe, on top of that, it only tiers memory when there’s memory pressure! So if you don’t see any tiering, it could simply be that you are not under any memory capacity pressure. (Why move pages to a lower tier when there’s no need?)

Please note, the below commands apply for all versions below vSphere/VCF 9.1! For VCF/vSphere 9.1 please look at this blog post!

List all storage devices via the CLI:

esxcli storage core device list

Create memory tiering partition on an NVMe device:

esxcli system tierdevice create -d=/vmfs/devices/disks/eui.1ea506b32a7f4454000c296a4884dc68

Enable Memory Tiering on a host level, note this requires a reboot:

esxcli system settings kernel set -s MemoryTiering -v TRUE

How is Memory Tiering configured in terms of DRAM to NVMe ratio? A 4:1 DRAM to NVMe ratio would be 25%, 1:1 would be 100%. So if you have it set at 4:1, with 512GB of DRAM you would only use 128GB of the NVMe at most, regardless of the size of the device.

esxcli system settings advanced list -o /Mem/TierNvmePct

Is memory tiered or not? Find out all about it via memstats!

memstats -r vmtier-stats -u mb

Want to show a select number of metrics?

memstats -r vmtier-stats -u mb -s name:memSize:active:tier1Target:tier1Consumed:tier1ConsumedPeak:comnsumed

So what would the outcome look like when there is memory tiering happening? I removed a bunch of the metrics, just to keep it readable, “tier1” is the NVMe device, and as you can see each VM has several MBs worth of memory pages on NVMe right now.

 VIRTUAL MACHINE MEMORY TIER STATS: Wed Dec 17 15:29:43 2025
 -----------------------------------------------
   Start Group ID   : 0
   No. of levels    : 12
   Unit             : MB
   Selected columns : name:memSize:tier1Consumed

----------------------------------------
           name    memSize tier1Consumed
----------------------------------------
      vm.533611       4096            12
      vm.533612       4096            34
      vm.533613       4096            24
      vm.533614       4096            11
      vm.533615       4096            25
----------------------------------------
          Total      20480           106
----------------------------------------

Unexplored Territory Episode 087 – Microsoft on VMware VCF featuring Deji Akomolafe

· ·

For the last episode of 2024, I invited Deji Akomolafe to discuss running Microsoft workloads on top of VCF. I’ve known Deji for a long time, and if anyone is passionate about VMware and Microsoft technology, it is him. Deji went over the many caveats, and best practices when it comes to running for instance SQL on top of VMware VCF (or vSphere for that matter). NUMA, CPU Scheduling, latency sensitive settings, power settings, virtual disk controllers, just some of the things you can expect in this episode. You can listen to the episode on Spotify, Apple, or via the embedded player below.