Yellow Bricks

by Duncan Epping

vSAN

Doing maintenance on a Two-Node (Direct Connect) vSAN configuration

· ·

I was talking to a partner and customer last week at a VMUG. They were running a two node (direct connect) vSAN configuration and had some issues during maintenance which were, to them, not easy to explain. What they did is they placed the host which was in the “preferred fault domain” in to maintenance mode. After they placed that host in to maintenance mode the link between the two hosts for whatever reason failed. After they rebooted the host in the preferred host it connected back to the witness but at this point in time the connection between the hosts had not returned yet. This confused vSAN and that resulted in the scenario where the VMs in the secondary fault domain were powered off. As you can imagine an undesired effect.

This issue is solved in the near future in a new version of vSAN, but for those who need to do maintenance on a two-node (direct connect) configuration (or a full site maintenance in a stretched environment) I would highly recommend the following simple procedure. This will need to be done when doing maintenance on the host which is in the “preferred fault domain”:

  • Change the preferred fault domain
    • Under vSAN, click Fault Domains and Stretched Cluster.
    • Select the secondary fault domain and click the Mark Fault Domain as preferred for Stretched Cluster icon
  • Place the host in to maintenance mode
  • Do your maintenance

Fairly straight forward, but important to remember…

vSAN Adaptive Resync, what does it do?

· ·

I am starting to get some more questions about vSAN Adaptive Resync lately. This was introduced a while back, but is also available in the latest versions of vSAN through vSphere 6.5 Patch 02. As a result various folks have started to look at it and are starting to wonder what it is. Hopefully by now everyone understands what resync traffic is and when you see resync traffic. The easiest example of course is a host failure. If a host has failed and there’s sufficient disk space and there’s additional hosts available to make the impacted VMs compliant with their policy again then vSAN will resync the data.

Resync aims to finish the creation of these new components asap, simple reason for this is availability. The longer the resync takes, the longer you are at risk. I think that makes sense right? In some cases however it may occur that when VMs are very busy and resync is happening that VM observed latency goes through the roof. We already had a manual throttling mechanism for when this situation occurs, but of course preferably vSAN should throttle resync traffic properly for you. This is what vSAN Adaptive Resync does.

So how does that work? Well, when the high watermark is reached for VM latency then vSAN will cut the bandwidth of resync in half. Next vSAN will check if the VM latency is below the low watermark, if not then it will cut resync traffic in half again. It does this until the latency is below the low watermark. When the latency is below the low watermark then vSAN will increase the bandwidth of resync traffic granularly until the low watermark is reached and stay at that level. (Some official info can be found in this kb, and this virtual blocks blog.)

Hope that helps,

Where is the vSAN storage performance proactive test in vSphere 6.5 U1 patch 02?

· ·

I had some customers asking where the storage performance proactive test and the multicast proactive test was in the latest release of vSAN. In the past this is what the UI looked like when they would go to the Proactive Test section:

proactive test disappeared

But now it looks like this:

proactive test disappeared

What happened? Well, two tests have been removed. I guess most people will understand why the Multicast test has been removed, with the disappearance of Multicast in vSAN the test was not needed any longer. To be clear, if you are running vSAN in unicast mode the test will not show, if you are running in multicast mode however then of course the test will still be shown. But what about the Storage Performance Test?

We have noticed that most customers were using HCI Bench when doing benchmarks or using their own tooling (please don’t use legacy tools). Those who were using the proactive test often drew incorrect conclusions as it does not provide the flexibility a solution like HCI Bench offers. VMware felt that HCI Bench was a more suitable solution for doing benchmarks and this is definitely VMware’s recommended solution, as such the decision was made to focus on HCI Bench from a development perspective and deprecate the perf benchmark feature in the Proactive Tests section.

Can I use a custom TCP/IP stack for the vSAN Network?

· ·

I got this question today if you can use a custom TCP/IP stack for the vSAN Network, and I had this before. This question is usually asked by customers who have a stretched cluster as the stretched cluster forces them to create different routes in an L3 network. Especially with a large number of hosts this is cumbersome and error prone, unless you have everything automated end to end, but we are not all William Lam or Alan Renouf right 🙂

The answer is unfortunately: No, we do not support custom TCP/IP stacks for vSAN and there’s no vSAN TCP/IP stack either. The default TCP/IP stack has to be used, this is documented on storagehub.vmware.com in the excellent vSAN Networking guide which my colleagues Paudie and Cormac wrote. Of course for things like vMotion you can use the vMotion TCP/IP stack while using vSAN, that is fully supported.

Currently, vSphere does not include a dedicated TCP/IP stack for the vSAN traffic service nor the supportability for the creation of custom vSAN TCP/IP stack. To ensure vSAN traffic in Layer 3 network topologies leaves over the vSAN VMkernel network interface, administrators need to add the vSAN VMkernel network interface to the default TCP/IP Stack and define static routes for all of the vSAN cluster members.

Holiday gift: vSAN Essentials book available for free

· ·

Christmas is coming so Cormac and I figured we would do something special for everyone, after a long debate we decided to make the vSAN Essentials book available for free. Note that this is the “Essential Virtual SAN” book which was published by VMware Press / Pearson and is based on the 6.2 version of vSAN. The book however is still very relevant today, and of course we are considering doing an update of the content to either the latest release, or maybe even to an upcoming release. You can read the book online (which is what we recommend), but you can also download it as PDF, EPUB or MOBI format. Basically you can read it anywhere, anytime and using any device. Nice right!?!

We used the Gitbook platform to publish the book and decided to leverage the beta version of gitbook as it looks very clean and makes the content easy to read online. Also, I have used the gitbook platform in the past for the HA Deepdive, and I wanted to give back by beta testing their platform. Ah well, instead of rambling on, here’s the book:

vsan-essentials.com

If you find anything unusual, please leave a comment here. Hope you will enjoy it, and appreciates us (the authors) giving back to the community. If you do then I hope you will consider donating to charity, the amount doesn’t matter, all help is welcome! I personally support Hardcore Help Foundation, and I hope you will considering doing the same! A donation of 10€ will provide clean, safe water to a family for two years. They need your help to reach out to more families in need.