Yellow Bricks

by Duncan Epping

Storage

What’s new for vSAN in 9.1?

· · Leave a Comment

VMware Cloud Foundation 9.1 was just announced, and that means a lot of new functionality has been released. Some of the features you already knew about, others may come as a surprise. I know Pete Koehler has a whole series he is going to release, so I am just going to introduce a couple of features that I feel everyone should know about. Here’s a list of what was just announced:

  • Native S3 Object Storage
  • Cyber Recovery enhancements with Any to vSAN ESA, Seeding, Tag-Based VM selection and more
  • Auto-RAID
  • Global Deduplication and enhanced compression
  • QLC support
  • Mixed mode for Remote Datastores (ESA <-> OSA)
  • Enhanced Capacity Reporting
  • Resizing Shared VMDKs

Now, some of these capabilities I have been talking about for a while now at events, like Native S3 Object Storage, but it is probably still worth explaining what is announced. Let’s discuss a few of the above.

Native S3 Object Storage

This, in my opinion, is probably together with the Cyber Protection platform, the biggest feature that was announced for 9.1.  Most of you probably use some kind of S3 Object Storage platform as a backup destination, or you may have developers (and apps) directly accessing an S3 bucket for various reasons. S3 Object Storage use cases and total capacity have exploded over the last decade. So far, we (VMware/Broadcom) have always partnered with 3rd party vendors to deliver these S3 capabilities on top of VCF, but more and more customers have asked for a well-integrated solution that would come as part of VCF. With an upcoming patch release of 9.1 the tech preview of Native S3 Object Storage will be released.

Although the platform has been referred to as “vSAN Native Object Storage” in the past, I feel it is more native to VCF. Although the configuration can be completed entirely through the API and CLI, most customers will likely consume the solution through VCF Automation. VCF Automation will provide the ability to have tenants (organizations) create their own S3 Object Storage Service (or even multiple), and have many buckets per S3 Object Storage Service. This will provide the logical isolation you would expect from a multi-tenancy platform. As a Provider Admin you simply enable S3 Object Storage for a region, and then each tenant who has resources in that region assigned can consume it and create the service, and subsequently buckets, as shown below.

What’s new for vSAN in 9.1?


I will probably upload a demo soon and will record a podcast episode specifically on this topic.

Cyber Recovery

Recovery from a ransomware attack and protection against it, is a hot topic these days for every CIO. Just do a Google search, and you will find countless examples of companies losing millions as a result of production outages due to an attack. At Explore 2025, we showed the world what a sophisticated ransomware recovery platform would potentially look like, and with VCF 9.1 and Site Recovery / Cyber Recover, we are finally at the stage where we can say we have a solution to help you recover safely and efficiently fully on-prem!

With 9.1 not only do we provide the option to replicate from any storage platform efficiently to vSAN ESA and have deep snapshot chains, but we now also have the option to build an isolated recovery environment / clean room on-premises. This platform comes fully integrated with VCF, and provides an orchestrated workflow to recovery from a ransomware attack. On top of that, the platform integrates with an EDR solution like Carbon Black or CrowdStrike. to ensure recovered data is clean. Of course, it will also work with other EDRs, but it would just not have the automated scanning and cleaning just yet. I’ve had Jatin on the podcast not too long ago to explain all the benefits of the platform, and will be having another episode on this topic soon!

Global Dedupe and enhanced Compression

From an efficiency perspective, various new enhancements have been introduced. Global Deduplication was already part of vSAN ESA, but only available through a support request, as of 9.1 the feature is available for all customers right there in the UI. Along with Global Deduplication going GA, support for Encryption at Rest with Global Dedupe has also been added. For European customers, do note, support for stretched clusters is not there just yet, so some of you have to wait with enabling Global Dedupe. Besides Global Dedupe, a brand new compression algorithm has been added. In the past, compression was done using LZ4, going forward, compression will be done using zStandard. zStandard allows for better tuning, making it more capacity and cost (CPU) efficient, which should result in a higher compression ratio over time.

Auto-RAID and QLC

Last but not least, I should probably also briefly talk about QLC and Auto-RAID. QLC support is mainly intended for Cyber Recovery deployments. As you can imagine, this is the perfect use case for a lower-tier flash device, which provides high capacity at the cost of performance and endurance. This is something to definitely keep in mind, as these devices are definitely not intended to be used in regular production environment. As always, VMware will provide guidance in terms of what is supported and what not, and special Ready Node configurations will be created for Cyber Recovery specifically.

Auto-RAID is a feature that surprised me personally as well. I had heard about plans to develop a new RAID mechanism, but hadn’t realized it was going to ship in the 9.1 release already. This new storage policy option allows vSAN to make intelligent decisions around the to be used RAID configuration for each object based on the size of the cluster, and the features enabled on the cluster. In other words, if you have Stretched Cluster enabled, Auto-RAID will ensure your VMs are stretched and protect the VMs within a site accordingly, based on the number of hosts. If Auto-RAID is enabled, all 9.1 clusters can be managed using the same policy if you prefer vSAN to make the decisions for you! Why is this useful? Well, if the size of the cluster changes, or the cluster is (un)stretched, the Auto-RAID policy will automatically re-configure all associated VMs. This removes the risk of having VMs incorrectly configured and removes the administrative burden of having to make changes to a policy and re-apply it to all the VMs.

I have planned for a podcast recording with Pete Koehler later this week, so expect a brand new episode covering all the above (and more) dropping soon!

What happens after a Site Takeover when my failed sites come back online again?

· · Leave a Comment

I got a question after the previous demo: what would happen if, after a Site Takeover, the two failed sites came back online again? I completely ignored this part of the scenario so far, I am not even sure why. I knew what would happen, but I wanted to test it anyway to confirm that what engineering had described actually happened. For those who cannot be bothered to watch a demo, what happens when the two failed sites come back online again is pretty straightforward. The “old” components of the impacted VMs are discarded, vSAN will recreate the RAID configuration as specified within the associated vSAN Storage Policy, and then a full resync will occur so that the VM is compliant again with the policy. Let me repeat one part: a full resync will occur! So if you do a Site Takeover, I hope you do understand what the impact will be. A full resync will take time, of course, depending on the connection between the data locations.

Does a Site Takeover work with a 2-node configuration?

· · 3 Comments

I got a question last week if vSAN Site Takeover also works with a 2-node configuration, and my answer was: yes, it should work. However, I had never tested it, so I figured I would build a quick lab environment and see if I was right. I recorded the result, here it is! The demo is pretty straight forward, let me describe what you will see:

  • 2-node vSAN environment
  • 1 VM named “photon-001”
  • Photon-001 VM is “stretched” across both hosts and has a witness component on the witness host
  • Host “.245” and the witness will fail and the components on those hosts will go “absent”
  • Photon-001 VM becomes inaccessible
  • We run the site-takeover command, which will reconfigure the Photon-001 VM
  • The Photon-001 VM becomes available again and it automatically restarted

What do I do after a vSAN Stretched Cluster Site Takeover?

· · 4 Comments

Over the last couple of months, various new vSAN features were announced. Two of those features are around the Stretched Cluster configuration, and have probably been the number 1 feature request for a few years. Now that we have Site Takeover and Site Maintenance functionality available, I am starting to get some questions about the impact of them, and in particular, the Site Takeover functionality is raising some questions.

For those who don’t know what these features are, let me describe them briefly:

Site Maintenance = The ability to place a full vSAN stretched cluster Fault Domain into maintenance mode at once. This ensures that all hosts within the fault domain have consistently stored the data, and all hosts will go into maintenance mode at the same time.

Site Takeover = This provides the ability when a Witness and a Data Site has failed to bring back the remaining site through a command line interface. This will reconstruct the remaining “site local” RAID configuration, making the objects available again, which will then allow vSphere HA to restart the VMs.

Now, the question that the above typically raises is what happens to the Witness and the Data Site that failed when you do the Site Takeover? If you look at the VMs RAID configuration, you will notice that both the Witness and the Data Site components of the sites that failed will completely disappear from the RAID configuration.

But what do you do next, because even after you run the Site Takeover, you still see your hosts and the witness in vCenter Server, and you still see a stretched cluster configuration in the UI. Now at first I thought that if the environment was completely up and running again, you had to go through some manual effort to reconstruct the stretched cluster. Basically, remove the failed hosts, wipe the disks, and recreate the stretched cluster. This is, however, not the case.

In the example above, if the Preferred site and the Witness site return for duty, vSAN will automatically discard the stale components in those previously failed sites. It will recreate new components for all objects, and it will do a full resync of the data.

If you end up in a situation where your hosts are completely gone (let’s say as a result of a fire), then you will have to do some kind of manual cleanup as follows, before you rebuild and add hosts back:

  • Remove the failed hosts from the vCenter inventory
  • Remove the witness from the vCenter inventory
    • Delete the witness from the vCenter Server it is running, a real delete!
  • Delete the surviving Fault Domain, this should be the only Fault Domain still listed in the vCenter interface
  • You now have a normal cluster again
  • Rebuild hosts and recreate the stretched cluster

I hope that helps,

vSAN Stretched Cluster vs Fault Domains in a “campus” setting?

· · 2 Comments

I got this question internally recently: Should we create a vSAN Stretched Cluster configuration or create a vSAN Fault Domains configuration when we have multiple datacenters within close proximity on our campus? In this case, we are talking about less than 1ms latency RTT between buildings, maybe a few hundred meters at most. I think it is a very valid question, and I guess it kind of depends on what you are looking to get out of the infrastructure. I wrote down the pros and cons, and wanted to share those with the rest of the world as well, as it may be useful for some of you out there. If anyone has additional pros and cons, feel free to share those in the comments!

vSAN Stretched Clusters:

  • Pro: You can replicate across fault domains AND protect additionally within a fault domain with R1/R5/R6 if required.
  • Pro: You can decide whether VMs should be stretched across Fault Domains or not, or just protected within a fault domain/site
  • Pro: Requires less than 5MS RTT latency, which is easily achievable in this scenario
  • Con/pro: you probably also need to think about DRS/HA groups (VM-to-Host)
  • Con: From an operational perspective, it also introduces a witness host, and sites, which may complicate things, and at the various least requires a bit more thinking
  • Con: Witness needs to be hosted somewhere
  • Con: Limited to 3 Fault Domains (2x data + 1x witness)
  • Con: Limited to 20+20+1 configuration

vSAN Fault Domains:

  • Pro: No real considerations around VM-to-host rules usually, although you can still use it to ensure certain VMs are spread across buildings
  • Pro: No Witness Appliance to manage, update or upgrade. No overhead of running a witness somewhere
  • Pro: No design considerations around “dedicated” witness sites and “data site”, each site has the same function
  • Pro: Can also be used with more than 3 Fault Domains or Datacenters, so could even be 6 Fault Domains, for instance
  • Pro: Theoretically can go up to 64 hosts
  • Con: No ability to protect additionally within a fault domain
  • Con: No ability to specify that you don’t want to replicate VMs across Fault Domains
  • Con/Pro: Requires sub-1ms RTT latency at all times, which is low, but will be achievable in a campus cluster, usually