Yellow Bricks

by Duncan Epping

vSAN ESA is using more CPU cycles than vSAN OSA?

· · 7 Comments

Over the last couple of weeks, I’ve had conversations with customers and partners who have been running performance benchmarks against both vSAN ESA and vSAN OSA. As you can imagine, people want to compare version 8 of OSA against version 8 of ESA, and that is completely fair. What I noticed though is that some of those customers came back with comments around CPU usage of vSAN OSA against ESA. The general comment we get is that vSAN ESA is using more CPU cycles than vSAN OSA.

When looking at it from a total number point of view, or CPU cycles consumed, it is very likely you will see vSAN ESA using more cycles than vSAN OSA. The question then typically arises why that is the case, as VMware (the vSAN team) has been claiming that vSAN ESA is much more efficient than vSAN OSA. To be fair, it is much more efficient. For instance data services like checksumming, encryption, and compression have moved to the top of the stack (as shown below) resulting in the fact that we don’t have to compress/encrypt data 3/4/5/6 times but can do it once at the source and then send it over the network to the destination.

Still, it leaves the question, why is more CPU capacity used? The answer is simple, you are pushing much more IO. We’ve seen customers easily reaching 4x the number of IOPS with ESA than with OSA. Even though ESA is more efficient, if you are pushing 4x (or more) the amount of IO then you will need to remember that those additional IOs also come at a cost, and that cost is CPU cycles to process them. So when you make a comparison, please compare apples to apples, and not apples to oranges.

The last thing I want to add, and hopefully I can share some data in the future, the use of RDMA with vSAN 8 ESA seems to have a significant impact on CPU usage, as in lower the amount of CPU required to produce the same results (or better results). So it is worth considering RDMA for sure when adopting vSAN 8 ESA!

Disable the re-registering of HA disabled VMs on other hosts!

· · Leave a Comment

Years ago we had various customers that complained about the fact that they had VMs that were disabled for HA and that these VMs would not be re-registered when the host they were registered on would fail. I can understand why you would want the VMs to be re-registered as this makes it easier to power-on those VMs when a host has failed. If the VM would not be re-registered automatically, and the host to which it was registered has failed, you would have to manually register the VM and only then would you be able to power on that VM.

Now, it doesn’t happen too often, but there are also situations where certain VMs are disabled for HA restarts (or powered off) and customers don’t want to have those VMs to be re-registered as these VMs are only allowed to run on one particular host. In that particular case you can simply disable the re-registering of HA disabled VMs through the use of an advanced setting. The advanced setting for this, and the value to use, is the following:

das.reregisterRestartDisabledVMs - false

What can I change about a vSAN ESA Ready Node?

· · Leave a Comment

I’ve had half a dozen people asking about this over the past weeks, it really seems more and more people are at the point of adopting vSAN ESA (Express Storage Architecture. When they look at the various vSAN ESA Ready Node configurations what stands out is that the current list is limited in terms of server models and configurations. (https://vmwa.re/vsanesahcl)

The list is being updated every week, last week for instance Supermicro popped up as a Server vendor. Of course, Dell, HPe, and Lenovo had been on the list since day 1. When you select the vendor, the ready node type, and the model you will then have the option to select a number of things, but in most cases, you seem to be limited to “Storage Device” and “Number of Storage Devices”. This however does not mean you cannot change anything. A knowledge base article has been released which describes what you can, and cannot change when it comes to these configurations! The KB article is listed on the vSAN ESA VMware Compatibility Guide list, but somehow it seems people don’t always notice the link. (Yes, I have asked the team to make the link more obvious somehow.)

Now when you look at the KB it lists what you can change, and what the rules are when it comes to making changes. For instance, you can change the CPU, but only for the same or higher core count and the same or higher base clock speed. For memory, you can increase the amount, and the same applies to storage capacity for instance. For storage it is even a bit more specific, you need to use the same make/model, so basically if the ReadyNode configuration lists a P5600 of 1.6TB, you can swap it for a P5600 of 3.2TB. We recently (May 20th 2023) had a change in support, and we now support the change of device model/make, as long as you follow the other guidelines mentioned in the KB. For instance, you can swap an Intel device for a Samsung, but that Samsung would need to be supported by the OEM vendor and needs to be the same (or higher) performance and endurance class. And of course the device needs to be certified for vSAN ESA: http://vmwa.re/vsanesahclc. Anyway, if you are configuring a Ready Node for ESA, make sure to check the KB so that you make supported changes!

Cross connecting vSAN Datastores with HCI Mesh in vSAN 8 OSA

· · 4 Comments

Yesterday I had a discussion internally on Slack about a configuration for a customer. The customer had multiple locations and had the potential to create various clusters within locations, or across locations. Now, as most of you know, I have been talking a lot about stretched clusters for the past decade. However, stretched clusters are not the answer to every problem. Especially not in situations where you end up with unbalanced configurations, or where you lack the ability to place a witness in a third location. Customers tend to gravitate towards stretched clusters as it provides them resiliency, and pooling of all resources even though they are across locations.

With vSAN when you have multiple clusters, you also have multiple vSAN Datastores. Having that separation of resources is typically appreciated. However, in some cases, customers prefer the flexibility of movement with limited overhead. Sure if you have multiple clusters you can simply Storage vMotion VMs from source cluster to destination cluster, but it does mean you need to move ALL the data with the VM, where in some cases you may not care where the data resides.

This is where HCI Mesh comes into play. With HCI Mesh you have the ability to mount a vSAN Datastore. Meaning, you have a “client” and a “server” cluster, and the client mounts the server. Within our documentation on core.vmware.com this is demonstrated as follows:

If you look at this diagram then the top two clusters are “client” clusters and the bottom one is the “server” cluster. This basically means that the “client” cluster consumed the datastore capacity from the “server” cluster. The above diagram however resulted in a bit of confusion as it does not show a situation where your client cluster can simultaneously be a server cluster. This is something I want to point out. You can create a true “mesh” with HCI Mesh! If you have two clusters, let’s say Cluster A and Cluster B, then you can mount the vSAN Datastore from Cluster A to Cluster B and the datastore from Cluster B to Cluster A. This is fully supported, and works great, as demonstrated in the below two screenshots. I tested this with vSphere/vSAN 8.0 OSA, but it is also supported with vSAN 7.0 U1 and up. Do note, vSAN ESA today does not support HCI Mesh just yet, hopefully it will in the near feature!


So before you decide how to configure vSAN, please look at all the capabilities provided, write down your requirements, and see what helps solving the challenges you are facing while meeting those requirements (in a supported way)!