Yellow Bricks

by Duncan Epping

vSphere

Rebuilding failed disk in VSAN versus legacy storage

· ·

This is one of those questions that comes up every now and then, I have written about this before, but it never hurts to repeat some of it. The comment I got was around rebuild time of failed drives in VSAN, surely it takes longer than with a “legacy” storage system. The answer of course is: it depends (on many factors).

But what does it depend on? Well it depends on what exactly we are talking about, but in general I think the following applies:

With VSAN components (copies of objects, in other words copies of data) are placed across multiple hosts, multiple diskgroups and multiple disks. Basically if you have a cluster of lets say 8 hosts with 7 disks each and you have 200 VMs then the data of those 200 VMs will be spread across 8 hosts and 56 disks in total. If one of those 56 disks happens to fail then the data that was stored on that disk would need to be reprotected. That data is coming from the other 7 hosts which is potentially 49 disks in total. You may ask, why not 55 disks? Well because replica copies are never stored on the same hosts for resiliency purposes, look at the diagram below where a single object is split in to 2 data components and a witness, they are all located on different hosts!

We do not “mirror” disks, we mirror the data itself, and the data can and will be place anywhere. This means that when a failure has occurred of a disk within a diskgroup on a host all remaining disk groups / disk / hosts will be helping to rebuild the impacted data, which is 49 disks potentially. Note that not only will disks and hosts containing impacted objects help rebuilding the data, all 8 hosts and 55 disks will be able to receive the replica data!

Now compare this to a RAID set with a spare disk. In the case of a spare disk you have 1 disk which is receiving all the data that is being rebuild. That single disk can only take an X number of IOPS. Lets say it is a really fast disk and it can take 200 IOPS. Compare that to VSAN… Lets say you used really slow disks which only do 75 IOPS… Still that is (potentially) 49 disks x 75 IOPS for reads and 55 disks for writes.

That is the major difference, we don’t have a single drive as a designated hot spare (or should I say bottleneck?), we have the whole cluster as a hot spare! As such rebuild times when using similar drives should always be faster with VSAN compared to traditional storage. 

SMP-FT and (any type of) stretched storage support

· ·

I had a question today around support for SMP-FT in an EMC VPLEX environment. It is well known that SMP-FT isn’t supported in a stretched VSAN environment, but what about other types of stretched storage? Is that a VSAN specific constraint? (Legacy) FT appears to be supported for VPLEX and other types of stretched storage?

SMP-FT is not supported in a vSphere Metro Storage Cluster environment either! This has not been qualified yet, I’ve requested the FT team to at least put it up on the roadmap and document max latency tolerated for these types of environments for SMP-FT just in case someone would want to use it in a campus situations for instance, despite the high bandwidth requirements for SMP-FT. Note that “legacy FT” can be used with vMSC environment, but not with VSAN. In order to use legacy FT (single vCPU) you will need to use an advanced VM setting: vm.uselegacyft. Make sure to set this setting when using FT in a stretched environment!

Jumbo Frames and VSAN Stretched Cluster configurations

· ·

I received a question last week from a customer who had implemented a stretched VSAN cluster. The Health Check after the implementation indicated that there was an “issue” with the MTU configuration. The customer had explained that he had configured an MTU of 9000 between the two data sites and an MTU of (default) 1500 between data sites and the witness.

The question of course was, why the Health Check indicated there was an issue. The problem here is that witness traffic and data in todays version of Virtual SAN use the same VMkernel interface. If the VSAN VMkernel interface on the the “data” site is configured for 9000 and one the “witness” site is configured for 1500 then there is a mismatch which causes fragmentation etc. This is what the health check calls out. VSAN (and the health check as such) expects an “end-to-end” consistently configured MTU, even in a stretched environment.

VSAN VROps Management Pack version 6.0.3 available

· ·

On the 15th the VROps Management Pack for VSAN 6.0.3 was released. If you have VROps Standard or higher you can take advantage of this management pack. It is supported for the latest release of VSAN, 6.1, as of this management pack officially. Very useful to find out if there are any anomalies and what the trends are. I’ve always loved VROps and it just became even more useful to me!

For those who want even more info, there is also a Log Insight Content Pack for VSAN available, which can give you some great insights on what is going on within your VSAN environment. For instance when there is congestion as shown in the screenshot below, which I borrowed from Cormac.

Removing stretched VSAN configuration?

· ·

I had a question today around how to safely remove a stretched VSAN configuration without putting any of the workloads in danger. This is fairly straight forward to be honest, there are 1 or 2 things though which are important. (For those wondering why you would want to do this, some customers played with this option and started loading workloads on top of VSAN and then realized it was still running in stretched mode.) Here are the steps required:

  1. Click on your VSAN cluster and go to Manage and disable the stretched configuration
    • This will remove the witness host, but will leave 2 fault domains in tact
  2. Remove the two remaining fault domains
  3. Go to the Monitor section and click on Health and check the “virtual san object health”. Most likely it will be “red” as the “witness components” have gone missing. VSAN will repair this automatically by default in 60 minutes. We prefer to take step 4 though asap after removing the failure domains!
  4. Click “repair object immediately”, now witness components will be recreated and the VSAN cluster will be healthy again.
  5. Click “retest” after a couple of minutes

By the way, that “repair object immediately” feature can also be used in the case of a regular host failure where “components” have gone absent. Very useful feature, especially if you don’t expect a host to return any time soon (hardware failure for instance) and have the spare capacity.