VMware Cloud Foundation

vSAN ESA Witness memory and CPU resources?

Duncan Epping · Mar 10, 2026 · Leave a Comment

Not sure when this happened, but somehow the resource requirements for the vSAN Witness VM disappeared. Someone asked me last week how much memory is allocated to a VM, and how many vCPUs. Now, of course, this depends on the profile you select as the Witness VM has an M, L, and XL profile. The profile you select is determined by the number of VMs you will be provisioning, yes it is smart to take a growth factor into account. Now, when you deploy the VM, it doesn’t give a hint either, but you can figure out the size by simply looking at the OVF descriptor file. So this is what I got from the vSAN ESA Witness OVF:

vSAN ESA Witness XL – 8 vCPUs – 64 GB memory
vSAN ESA Witness L – 4 vCPUs – 32 GB memory
vSAN ESA Witness M – 4 vCPUs – 16 GB memory

And for those who were wondering, with vSAN OSA the requirements are:

vSAN OSA Witness XL – 6 vCPUs – 32 GB memory
vSAN OSA Witness L – 2 vCPUs – 32 GB memory
vSAN OSA Witness Normal – 2 vCPUs – 16 GB memory
vSAN OSA Witness Tiny – 2 vCPUs – 8 GB memory

I hope that helps, and also please do note… if you read this article a few years from now, things may have changed!

Can I replicate, or snapshot, my vSAN Stretched Cluster Witness appliance for fast recovery?

Duncan Epping · Jan 20, 2026 · Leave a Comment

I’ve been seeing this question pop up more frequently, can I replicate or snapshot my vSAN Stretched Cluster Witness appliance for fast recovery? Usually, people ask this question as they cannot adhere to the 3-site requirement for a vSAN Stretched Cluster. So by setting up some kind of replication mechanism with low RPO, they try to mitigate this risk.

I guess the question stems from a lack of understanding of what the witness does. The witness provides a quorum mechanism, the quorum mechanism helps determine which site has access to the data in the case of a network failure (ISL) between the data locations.

So why can the Witness Appliance not be snapshotted or replicated then? Well, in order to provide this quorum mechanism, the Witness Appliance stores a witness component for each object. This is not per site, or per VM, but for every object… So if you have a VM with multiple VMDKs, you will have multiple witness objects per VM stored on the witness appliance. That witness object holds metadata and, through a log sequence number, understands which object holds the most recent data. This is where the issue arises. If you revert a Witness Appliance to an earlier point in time, the witness components also revert to an earlier point in time, and will have a different log sequence number than expected. This results in vSAN being unable to make the object available to the surviving site, or the site that is expected to hold quorum.

So in short, should you replicate or snapshot the Witness Appliance? No!

Playing around with Memory Tiering, are my memory pages tiered?

Duncan Epping · Dec 18, 2025 · 1 Comment

There was a question on VMTN about Memory Tiering performance, and how you can check if pages were tiered. I haven’t played around with Memory Tiering too much, so I noted down for myself what I needed to do on every host in order to enable it. Note, if the command contains a path and you want to do this in your own environment you need to change the path and device name accordingly. The question was if memory pages were tiered or not, so I dug up the command that allows you to check this on a per host level. It is at the bottom of this article for those who just want to skip to that part.

Now, before I forget, probably worth mentioning as this is something many people don’t seem to understand, memory tiering only tiers cold memory pages. Active pages are not being moved to NVMe, on top of that, it only tiers memory when there’s memory pressure! So if you don’t see any tiering, it could simply be that you are not under any memory capacity pressure. (Why move pages to a lower tier when there’s no need?)

List all storage devices via the CLI:

esxcli storage core device list

Create memory tiering partition on an NVMe device:

esxcli system tierdevice create -d=/vmfs/devices/disks/eui.1ea506b32a7f4454000c296a4884dc68

Enable Memory Tiering on a host level, note this requires a reboot:

esxcli system settings kernel set -s MemoryTiering -v TRUE

How is Memory Tiering configured in terms of DRAM to NVMe ratio? A 4:1 DRAM to NVMe ratio would be 25%, 1:1 would be 100%. So if you have it set at 4:1, with 512GB of DRAM you would only use 128GB of the NVMe at most, regardless of the size of the device.

esxcli system settings advanced list -o /Mem/TierNvmePct

Is memory tiered or not? Find out all about it via memstats!

memstats -r vmtier-stats -u mb

Want to show a select number of metrics?

memstats -r vmtier-stats -u mb -s name:memSize:active:tier1Target:tier1Consumed:tier1ConsumedPeak:comnsumed

So what would the outcome look like when there is memory tiering happening? I removed a bunch of the metrics, just to keep it readable, “tier1” is the NVMe device, and as you can see each VM has several MBs worth of memory pages on NVMe right now.

 VIRTUAL MACHINE MEMORY TIER STATS: Wed Dec 17 15:29:43 2025
 -----------------------------------------------
   Start Group ID   : 0
   No. of levels    : 12
   Unit             : MB
   Selected columns : name:memSize:tier1Consumed

----------------------------------------
           name    memSize tier1Consumed
----------------------------------------
      vm.533611       4096            12
      vm.533612       4096            34
      vm.533613       4096            24
      vm.533614       4096            11
      vm.533615       4096            25
----------------------------------------
          Total      20480           106
----------------------------------------

What do I do after a vSAN Stretched Cluster Site Takeover?

Duncan Epping · Nov 10, 2025 · 4 Comments

Over the last couple of months, various new vSAN features were announced. Two of those features are around the Stretched Cluster configuration, and have probably been the number 1 feature request for a few years. Now that we have Site Takeover and Site Maintenance functionality available, I am starting to get some questions about the impact of them, and in particular, the Site Takeover functionality is raising some questions.

For those who don’t know what these features are, let me describe them briefly:

Site Maintenance = The ability to place a full vSAN stretched cluster Fault Domain into maintenance mode at once. This ensures that all hosts within the fault domain have consistently stored the data, and all hosts will go into maintenance mode at the same time.

Site Takeover = This provides the ability when a Witness and a Data Site has failed to bring back the remaining site through a command line interface. This will reconstruct the remaining “site local” RAID configuration, making the objects available again, which will then allow vSphere HA to restart the VMs.

Now, the question that the above typically raises is what happens to the Witness and the Data Site that failed when you do the Site Takeover? If you look at the VMs RAID configuration, you will notice that both the Witness and the Data Site components of the sites that failed will completely disappear from the RAID configuration.

But what do you do next, because even after you run the Site Takeover, you still see your hosts and the witness in vCenter Server, and you still see a stretched cluster configuration in the UI. Now at first I thought that if the environment was completely up and running again, you had to go through some manual effort to reconstruct the stretched cluster. Basically, remove the failed hosts, wipe the disks, and recreate the stretched cluster. This is, however, not the case.

In the example above, if the Preferred site and the Witness site return for duty, vSAN will automatically discard the stale components in those previously failed sites. It will recreate new components for all objects, and it will do a full resync of the data.

If you end up in a situation where your hosts are completely gone (let’s say as a result of a fire), then you will have to do some kind of manual cleanup as follows, before you rebuild and add hosts back:

Remove the failed hosts from the vCenter inventory
Remove the witness from the vCenter inventory
- Delete the witness from the vCenter Server it is running, a real delete!
Delete the surviving Fault Domain, this should be the only Fault Domain still listed in the vCenter interface
You now have a normal cluster again
Rebuild hosts and recreate the stretched cluster

I hope that helps,

vSAN Stretched Cluster vs Fault Domains in a “campus” setting?

Duncan Epping · Sep 25, 2025 · 2 Comments

I got this question internally recently: Should we create a vSAN Stretched Cluster configuration or create a vSAN Fault Domains configuration when we have multiple datacenters within close proximity on our campus? In this case, we are talking about less than 1ms latency RTT between buildings, maybe a few hundred meters at most. I think it is a very valid question, and I guess it kind of depends on what you are looking to get out of the infrastructure. I wrote down the pros and cons, and wanted to share those with the rest of the world as well, as it may be useful for some of you out there. If anyone has additional pros and cons, feel free to share those in the comments!

vSAN Stretched Clusters:

Pro: You can replicate across fault domains AND protect additionally within a fault domain with R1/R5/R6 if required.
Pro: You can decide whether VMs should be stretched across Fault Domains or not, or just protected within a fault domain/site
Pro: Requires less than 5MS RTT latency, which is easily achievable in this scenario
Con/pro: you probably also need to think about DRS/HA groups (VM-to-Host)
Con: From an operational perspective, it also introduces a witness host, and sites, which may complicate things, and at the various least requires a bit more thinking
Con: Witness needs to be hosted somewhere
Con: Limited to 3 Fault Domains (2x data + 1x witness)
Con: Limited to 20+20+1 configuration

vSAN Fault Domains:

Pro: No real considerations around VM-to-host rules usually, although you can still use it to ensure certain VMs are spread across buildings
Pro: No Witness Appliance to manage, update or upgrade. No overhead of running a witness somewhere
Pro: No design considerations around “dedicated” witness sites and “data site”, each site has the same function
Pro: Can also be used with more than 3 Fault Domains or Datacenters, so could even be 6 Fault Domains, for instance
Pro: Theoretically can go up to 64 hosts
Con: No ability to protect additionally within a fault domain
Con: No ability to specify that you don’t want to replicate VMs across Fault Domains
Con/Pro: Requires sub-1ms RTT latency at all times, which is low, but will be achievable in a campus cluster, usually