Yellow Bricks

by Duncan Epping

vsan esa

Scalable Snapshots demo with the vSAN 8.0 Express Storage Architecture

· · 1 Comment

Starting with vSAN 8 a brand new architecture was introduced called “Express Storage Architecture”. Over the last year or so a lot of information has been shared about ESA and the benefits of ESA. One of the things which ESA introduces is much-improved snapshot scalability.

With vSAN OSA, and with VMFS, when you create a snapshot you typically immediately see a performance degradation. This is because both VMFS and vSAN OSA still operate using the redo-log based snapshot mechanism. This means that with vSAN OSA when you create a snapshot a new object is created and writes are re-directed. It also means that reads will be coming from various files, if you have one or more snapshots. This mechanism is, unfortunately, not very effective. Let me borrow a diagram that is part of a post John Nicholson wrote to demonstrate that old logic.

With vSAN 8 ESA the mechanism has changed and no longer does vSAN, or vSphere for that matter, create an additional object. vSAN ESA handles this on a meta-data level. In other words, instead of redirecting writes and traversing files for reads, vSAN now leverages a highly efficient B-Tree structure and pointers to keep track of which block is associated with which snapshot.

Not only is this more efficient from a capacity perspective, but more importantly it is very efficient from a performance standpoint. I ran half a dozen tests in my lab, and what I saw was a below 2% performance impact between a VM without a snapshot and a VM with one or multiple snapshots. I could NOT see a significant difference between the first or the fifth snapshot. I do want to point out that my lab is not officially certified to run vSAN ESA, nevertheless, I was very impressed with the results.

During the last run, I actually recorded the whole exercise. In this demo, I show the creation of one snapshot, while the VM is running a benchmark (HCIBench). Now, during the testing, I created not one but various snapshots and of course, I deleted all of them as well. You have all probably experienced extensive stun times during the deletion of a snapshot at times, and this is where vSAN ESA shines. The stun times have been reduced by 100 times, and that is something I am sure each of you will appreciate. Why have they been reduced drastically? Well, simply because we no longer have to copy data from one vSAN object to another. This makes a huge difference, not just for stun times, but also for performance in general (latency, IOPS, throughput). If you are interested, have a look at the demo!

MAXimizing vSAN’s potential with the Express Storage Architecture (vSAN Max)

· · 2 Comments

Last week at VMware Explore a few vSAN features and offerings were announced, one of them being vSAN Max! All week I have been having conversations with customers who were highly excited about the new solution. For those who did not read the announcements, or listened to the Unexplored Territory Podcast episode on the topic, let me go over what was announced and what vSAN Max is.

As most of you know, vSAN is a hyperconverged storage platform delivered via VMware’s flagship product vSphere. This means that if you have vSphere running, vSAN is literally two clicks away from being enabled. You will need local storage devices, and those local devices then will be formed into a shared datastore on top of which you can run your VMs. Although HCI solutions work for most customers, at certain levels of scale it may be preferred to have a disaggregated solution and share a dedicated storage platform with one or multiple vSphere clusters. This is what vSAN Max brings to the table.

Looking at the above diagram a few things stand out when it comes to vSAN Max. First of all, it says “Storage Only” and secondly it mentions “Supports high-density ESA ReadyNodes”. There are a few things to unwrap here. Firstly, vSAN Max is based on vSAN Express Storage Architecture, aka vSAN ESA. This means that it is a single tier of storage, based on NVMe flash devices. On top of that, it also means that all available data services will also be available on vSAN Max: Fault Domains, Stretched Clustering, vSAN File Services, iSCSI, Compression, Encryption etc. All of these are also included by default in the license by the way, it is just a single edition from a licensing point of view and it will include vSphere. In other words, vSphere + vSAN Enterprise by default, and licensed on capacity instead of CPU/Cores.

Secondly, it mentions “high-density”, vSAN Max starts at 200TB per host, and has a minimum of 6 hosts per cluster. This means that the starting capacity is 1.2 Petabytes for a vSAN Max cluster. The maximum number of hosts within a cluster is 32 at the time of writing (but 24 hosts being the recommended maximum), and it will support up to 8.6 Petabytes and around 3.4 million IOPS.

It also mentions ReadyNodes, and let me stress this, ReadyNodes! We still see a lot of customers picking random components for their vSAN cluster and then being surprised that Skyline Health reports the cluster is not supported. For vSAN Max there will be a separate set of vSAN ReadyNode configurations. These configurations will have for instance 100Gbps network cards, and as mentioned a minimum of 200TB per host.

Now, this doesn’t mean that the connecting clusters need to be running 100GbE, they can be even 1Gbps connected, that’s up to you and the requirements you have from a performance perspective. The 100GbE connections will be used for intra-cluster communications, so the switching architecture also needs to cater to this.

Knowing all of this, you may wonder what the use cases are for vSAN Max. As Pete Koehler mentioned, it can be used for anything, but is primarily targeted at those with high capacity requirements and who prefer a centralized model, but still want to manage their storage platform through vCenter Server and use all the bells and whistles that come with it (and with VROps for instance).

Hopefully, that provides some insights in terms of what to expect when vSAN Max goes “general availability” I will follow up with some short demos showing what it will look like, although that will probably be relatively boring as it will look very similar to vSAN ESA. In the meanwhile, there’s a bunch of material on the VMware website that you can check out.

New book: VMware vSAN 8.0 U1 Express Storage Architecture Deep Dive!

· · 9 Comments

We already gave some hints on twitter, and during an episode of the Unexplored Territory podcast, but here it finally is… The new book, the VMware vSAN 8.0 U1 Express Storage Architecture Deep Dive! It has been a year since we released the vSAN 7.0 U3 Deep Dive book, and with this brand new vSAN architecture being introduced in vSAN 8.0 we figured it was time to do a full overhaul of the book as well. Mind you, this new book purely deals with the Express Storage Architecture, aka vSAN ESA. This also means that some of the features which are not supported by ESA are not discussed in this book, for that you will need to buy the vSAN 7.0 U3 Deep Dive book, which covers OSA. Another big change is that we brought in a third author, we asked our good friend Pete Koehler to contribute to the book. Pete had done reviews of previous books, and considering the amount of material he produced for VMware Tech Marketing for vSAN (and ESA specifically) it made a lot of sense to bring him in!

VMware’s vSAN has rapidly proven itself in environments ranging from hospitals to oil rigs to e-commerce platforms and is the market leader in the hyperconverged space. Along the way, the world of IT has rapidly changed, not just from a software point of view, but also from a hardware perspective. With vSAN 8.0 VMware brought a new architecture to market called vSAN Express Storage Architecture (ESA). This architecture is highly optimized for today’s world of datacenter resources, be it CPU, memory, networking, or NVMe based flash storage.

The authors of the vSAN Deep Dive have thoroughly updated their definitive guide to this transformative technology. Writing for vSphere administrators, architects, and consultants, Cormac Hogan, Duncan Epping , and Pete Koehler explain what vSAN ESA is, why the architecture has changed, what it now offers, and how to gain maximum value from it. The book offers expert insight into preparation, installation, configuration, policies, provisioning, clusters, architecture, and more. You’ll also find practical guidance for using all data services, stretched clusters, two-node configurations, and cloud-native storage services.

Although we pressed publish on Tuesday, sometimes it takes a while before the book is available in all Amazon stores, but it should just trickle down in the upcoming 24-48 hours. The book is priced at 9.99 USD for the ebook and 29.99 USD for a paper copy, and is sold through Amazon only. Get it while it is hot, and we would appreciate it if you would use our referral links and leave a review when you finish it. Thanks for the support, and we hope you will enjoy it!

Of course, we also have the links to other major Amazon stores:

vSAN ESA ReadyNode configurations are more flexible than you think!

· · 4 Comments

I had a discussion at the Dutch VMUG yesterday about the ReadyNode configurations for vSAN ESA. The discussion was about how difficult it was to select a host and customize it. It was then that I realized that most people hadn’t noticed yet that there is an easier method (or lifehack as my kids would say) when it comes to selecting your server model. How does that work? Well, let me show you!

First, let’s take a look at the vSAN ESA ReadyNode Hardware Guidance Table. The table below shows you what the node capacity is for each profile from a storage, CPU, memory, and networking perspective.

Now if you look at the table you will see that as the “profile” number goes up, so does the capacity for each of the various components. This is actually what provides you with a lot of flexibility in my opinion. If we take Dell as an example, but the same applies for most vendors on the current list, and we select “vSAN-ESA-AF2” and look at the list of options we see the following:

  • PowerEdge R650
  • PowerEdge R6515
  • PowerEdge R750
  • PowerEdge R7515

Now, if we look at “vSAN-ESA-AF8” next, which is the highest profile, we see that we only can pick 1 server model, which happens to be the PowerEdge R750. If we then look at the difference between the hosts selected for each profile a few things stand out:

  • vSAN-ESA-AF2 has an Intel Xeon Silver 4314, while vSAN-ESA-AF8 has a Platinum 8358
  • vSAN-ESA-AF2 has 512GB, while vSAN-ESA-AF8 has 1024GB
  • vSAN-ESA-AF2 a 25Gbps NIC, while vSAN-ESA-AF8 has a 100Gbps NIC
  • vSAN-ESA-AF2 has five 3.2TB NVMe devices while vSAN-ESA-AF8 has twenty-four 3.2TB devices

Now if I look at the KB article which explains what you can, and cannot change, something stands out, most of the components can be modified/customized. For instance, for CPU you can go to a higher core count and/or higher base clock speed! For memory, you can go up, same for storage devices (as long as you stay within supported limits), etc etc.

In other words, what is the difference between a vSAN-ESA-AF2 and a vSAN-ESA-AF8? Basically the expected workload, the performance, the capacity. This ultimately results in a different configuration. Nothing, at this point in time, stops you from selecting the “lowest” vSAN ReadyNode Profile and spec it as an “AF4”, “AF6” or “AF8” from a CPU stance, or from a storage/memory capacity point of view. If you want to have some more flexibility, try selecting a smaller profile, select the host type, and increase the resources/components where needed!

When you start exploring the options it may seem complex, but when you look more closely you will quickly realize that it actually isn’t that complex, and that it actually provides you with a lot of flexibility, as long as you stick to the rules and pick supported components!

What can I change about a vSAN ESA Ready Node?

· · 2 Comments

I’ve had half a dozen people asking about this over the past weeks, it really seems more and more people are at the point of adopting vSAN ESA (Express Storage Architecture. When they look at the various vSAN ESA Ready Node configurations what stands out is that the current list is limited in terms of server models and configurations. (https://vmwa.re/vsanesahcl)

The list is being updated every week, last week for instance Supermicro popped up as a Server vendor. Of course, Dell, HPe, and Lenovo had been on the list since day 1. When you select the vendor, the ready node type, and the model you will then have the option to select a number of things, but in most cases, you seem to be limited to “Storage Device” and “Number of Storage Devices”. This however does not mean you cannot change anything. A knowledge base article has been released which describes what you can, and cannot change when it comes to these configurations! The KB article is listed on the vSAN ESA VMware Compatibility Guide list, but somehow it seems people don’t always notice the link. (Yes, I have asked the team to make the link more obvious somehow.)

Now when you look at the KB it lists what you can change, and what the rules are when it comes to making changes. For instance, you can change the CPU, but only for the same or higher core count and the same or higher base clock speed. For memory, you can increase the amount, and the same applies to storage capacity for instance. For storage it is even a bit more specific, you need to use the same make/model, so basically if the ReadyNode configuration lists a P5600 of 1.6TB, you can swap it for a P5600 of 3.2TB. We recently (May 20th 2023) had a change in support, and we now support the change of device model/make, as long as you follow the other guidelines mentioned in the KB. For instance, you can swap an Intel device for a Samsung, but that Samsung would need to be supported by the OEM vendor and needs to be the same (or higher) performance and endurance class. And of course the device needs to be certified for vSAN ESA: http://vmwa.re/vsanesahclc. Anyway, if you are configuring a Ready Node for ESA, make sure to check the KB so that you make supported changes!