Yellow Bricks

by Duncan Epping

Where did ESXi 6.5.0 build 7526125 go?

· ·

I had two customers asking today what happened to ESXi 6.5 build 7526125. They downloaded patches and installed them in their test environment. Ready to patch some of their clusters they did a validation and found out that the patch (ESXi650-201801001.zip) has disappeared from the face of the earth. This patch included microcode for Intel processors, and Intel informed VMware that there was potentially an issue with their microcode. As such VMware decided to pull the patch as noted in the KB article. Those who had already downloaded the patches and are manually updating, make sure to delete these. Those who use VUM, make sure to exclude them from your baseline as mentioned in the KB:

Any baseline (including VMware Pre-defined Baseline), that includes one or more of  the bulletins that  correspond to patch VMSA-2018-0004, would experience the above listed error and hence, will not be able to proceed with the remediation process. For such customers, it is recommended to create dynamic or static baseline excluding the bulletins ESXi650-201801401-BG, ESXi650-201801402-BG,  ESXi600-201801401-BG,  ESXi600-201801402-BG ,ESXi550-201801401-BG and continue with the remediation process. For more information on Create and Edit Patch or Extension Baselines see vSphere 6.5 document.

Normally I don’t share these types of things anymore, but as I had two people asking on the same day I figured I would as it seems not everyone had seen that the patches were pulled and replaced. If you haven’t downloaded the patches yet, or haven’t patched your systems but want to, read this advisory first and use the patches mentioned it.

Coming to a VMUG in EMEA near you…

· ·

I just came back from the VMUG in Denmark, which was a great event, and I figured I would share what I have planned right now for the upcoming months. I hope to be able to meet many of my readers at one of these events. If you are there, don’t be afraid to stop by and say hi! I am also more than happy to take back any feedback you have on storage/availability to our engineering and product management teams! (Oh and if you are a VMUG leader in EMEA and you would like to have my speak at one of your events, just drop me an email.)

Here we go:

  • 22 Feb – Newcastle, UK VMUG
  • 7 March – Lausanne, Switzerland VMUG
  • 8 March – Zurich, Switzerland VMUG
  • 20 March – Den Bosch, Netherlands VMUG
  • 10 April – Istanbul, Turkey VMUG

See you there!

 

vSAN Adaptive Resync, what does it do?

· ·

I am starting to get some more questions about vSAN Adaptive Resync lately. This was introduced a while back, but is also available in the latest versions of vSAN through vSphere 6.5 Patch 02. As a result various folks have started to look at it and are starting to wonder what it is. Hopefully by now everyone understands what resync traffic is and when you see resync traffic. The easiest example of course is a host failure. If a host has failed and there’s sufficient disk space and there’s additional hosts available to make the impacted VMs compliant with their policy again then vSAN will resync the data.

Resync aims to finish the creation of these new components asap, simple reason for this is availability. The longer the resync takes, the longer you are at risk. I think that makes sense right? In some cases however it may occur that when VMs are very busy and resync is happening that VM observed latency goes through the roof. We already had a manual throttling mechanism for when this situation occurs, but of course preferably vSAN should throttle resync traffic properly for you. This is what vSAN Adaptive Resync does.

So how does that work? Well, when the high watermark is reached for VM latency then vSAN will cut the bandwidth of resync in half. Next vSAN will check if the VM latency is below the low watermark, if not then it will cut resync traffic in half again. It does this until the latency is below the low watermark. When the latency is below the low watermark then vSAN will increase the bandwidth of resync traffic granularly until the low watermark is reached and stay at that level. (Some official info can be found in this kb, and this virtual blocks blog.)

Hope that helps,

Where is the vSAN storage performance proactive test in vSphere 6.5 U1 patch 02?

· ·

I had some customers asking where the storage performance proactive test and the multicast proactive test was in the latest release of vSAN. In the past this is what the UI looked like when they would go to the Proactive Test section:

proactive test disappeared

But now it looks like this:

proactive test disappeared

What happened? Well, two tests have been removed. I guess most people will understand why the Multicast test has been removed, with the disappearance of Multicast in vSAN the test was not needed any longer. To be clear, if you are running vSAN in unicast mode the test will not show, if you are running in multicast mode however then of course the test will still be shown. But what about the Storage Performance Test?

We have noticed that most customers were using HCI Bench when doing benchmarks or using their own tooling (please don’t use legacy tools). Those who were using the proactive test often drew incorrect conclusions as it does not provide the flexibility a solution like HCI Bench offers. VMware felt that HCI Bench was a more suitable solution for doing benchmarks and this is definitely VMware’s recommended solution, as such the decision was made to focus on HCI Bench from a development perspective and deprecate the perf benchmark feature in the Proactive Tests section.

Can I use a custom TCP/IP stack for the vSAN Network?

· ·

I got this question today if you can use a custom TCP/IP stack for the vSAN Network, and I had this before. This question is usually asked by customers who have a stretched cluster as the stretched cluster forces them to create different routes in an L3 network. Especially with a large number of hosts this is cumbersome and error prone, unless you have everything automated end to end, but we are not all William Lam or Alan Renouf right 🙂

The answer is unfortunately: No, we do not support custom TCP/IP stacks for vSAN and there’s no vSAN TCP/IP stack either. The default TCP/IP stack has to be used, this is documented on storagehub.vmware.com in the excellent vSAN Networking guide which my colleagues Paudie and Cormac wrote. Of course for things like vMotion you can use the vMotion TCP/IP stack while using vSAN, that is fully supported.

Currently, vSphere does not include a dedicated TCP/IP stack for the vSAN traffic service nor the supportability for the creation of custom vSAN TCP/IP stack. To ensure vSAN traffic in Layer 3 network topologies leaves over the vSAN VMkernel network interface, administrators need to add the vSAN VMkernel network interface to the default TCP/IP Stack and define static routes for all of the vSAN cluster members.