Yellow Bricks

by Duncan Epping

vSAN

vSAN 7.0 u3: IO Trip Analyzer

· · 1 Comment

In vSAN 7.0 U1 a new feature was introduced called IO Insight. IO Insight basically enabled customers to profile workloads and it provided them with a lot of information around the type of IO the workload was producing. In vSAN 7.0 U3 this is taken one step further with the IO Trip Analyzer. The IO Trip Analyzer provides details around the, yes you guessed it, trip of the IO. It basically informs you about the latency introduced at the various stages of the path the IO has to travel to end up on the capacity layer of vSAN.

Why would you need this? This tool is going to be very useful and will complement IO Insight when it comes to doing performance troubleshooting, or when it comes to getting a better understanding of the IO path. IO Trip Analyzer can be easily enabled by going to the “Monitor” section of the VM you want to enable it for.

You simply click “Run new test” and then specify for how long you want to analyze the VM (between 5 and 60 minutes). When the specified amount has passed you simply click on “View Result” and this will then provide you a diagram of the VM and its components.

When you then click on one of the dots, you will be able to see what kind of latency is introduced on the layer. It will provide you a potential cause for the latency and it will provide you some insights in terms of how you can potentially resolve the latency. Also, if there’s a significant amount of latency introduced then of course the diagram will show this through colors for the respective layer where the latency is introduced.

Before I share the demo, I should probably mention that there are some limitations in this first release of the IO Trip Analyzer (Not supported: Stretched Clusters, CNS persistent volumes, iSCSI etc) but I suspect those limitations will be lifted with every follow-up release of vSAN. I truly feel that this is a big improvement when it comes to performance troubleshooting, and I can’t wait to see what the vSAN team is planning for the future of this release.

Booting ESXi from SD/USB devices? Time to reconsider when buying new hardware!

· · 26 Comments

We’ve all seen those posts from people about worn-out SD/USB devices, or maybe even experience it ourselves at some point in time. Most of you reading this probably also knew there was an issue with 7.0 U2, which resulted in USB/SD devices wearing out a lot quicker. Those issues have been resolved with the latest patch for 7.0 U2. It has, however, resulted in a longer debate around whether SD/USB devices should still be used for booting ESXi, and it seems that the jury has reached a verdict.

On the 16th of September, a KB article was published by VMware, which contains statements around the future of SD/USB devices. I can be short about it, if you are buying new hardware make sure to have a proper persistent storage device, USB/SD is not the right choice going forward! Why? The volume of reads/writes to and from the OS-DATA partition continues to increase with every release, which means that the lower grade devices will simply wear out faster. Now, I am not going to repeat word for word what is mentioned in the KB, I would just like to urge everyone to read the KB article, and make sure to plan accordingly! Personally, I am a fan of M.2 flash devices for booting. They are not too expensive(greenfield deployments), plus they can provide enterprise-grade persistent storage to store all your ESXi related data. Make sure to follow the requirements around endurance though!

vSAN Storage Rules policy capability allows to set dedupe per VM?

· · 4 Comments

There was a question posted on the VMware Community Forums, and as this is something I have been asked regularly, I figured I would do a quick blog post about it. Although I have covered this before, it doesn’t hurt to repeat, as it appears to be somewhat confusing for people. When you create a VM Storage Policy, starting with vSAN 7.0 U2 you have the ability to specify if a VM needs to be Encrypted, have Dedupe and Compression enabled, have Compression-Only enabled, and/or needs to be stored on all-flash vSAN or Hybrid. Never noticed it? Look at the screenshot below.

In the screenshot, you see that you have the ability to specify which data service needs to be enabled. I guess this is where the confusion comes into play, as this functionality is not about enabling the data service for the VM to which you assign the policy. This is about which data service needs to be enabled on the datastore to which the VM can be provisioned. Huh, what? Okay, let’s explain.

If you are using vSAN as your storage platform, and you are sharing vSAN Datastores between clusters leveraging the HCI Mesh feature, then you could find yourself in a situation where some clusters are hybrid and some are all-flash. Some may have data services enabled like Encryption or Deduplication, some may not. In that scenario you want to be able to specify which features need to be enabled for the datastore the VM is provisioned to. So what this “storage rules” feature does is that it ensure that the datastore which is shown as “compatible” actually has the specified capabilities enabled! In other words, if you tick “data-at-rest encryption” in a policy and assign the policy to a VM, then only the datastores which have “data-at-rest encryption” enabled will be shown as compatible with your VM!

So again, “storage rules” apply to the data services that should be enabled on the vSAN Datastore, and do not enable data services on a per VM/VMDK basis.

HCI Mesh error: Failed to run the remote datastore mount pre-checks

· · 1 Comment

I had two comments on my HCI Mesh compute only blogpost where both were reporting the same error when trying to mount a remote datastore. The error that popped up was the following:

Failed to run the remote datastore mount pre-checks.

I tried to reproduce it in my lab, as both had upgraded from 7.0 to U2 I did the same, that didn’t result in the same error though. The error doesn’t provide any details around why the pre-check fails, as shown below in the screenshot. After some digging I found out that the solution is simple though, you need to make sure IPv6 is enabled on your hosts. Yes, even when you are not using IPv6, it still needs to be enabled to pass the pre-checks. Thanks, Jiří and Reza for raising the issue!

HCI Mesh error: Failed to run the remote datastore mount pre-checks

Does vSAN Enhanced Durability work when you have a limited number of hosts?

· · 3 Comments

Last week I had a question about how vSAN Enhanced Durability works when you have a limited number of hosts. In this case, the customer had a 3+3+1 stretched cluster configuration, and they wondered what would happen when they would place a host into maintenance mode. Although I was pretty sure I knew what would happen, I figured I would test it in the lab anyway. Let’s start with a high-level diagram of what the environment looks like. Note I use a single VM as an example, just to keep the scenario easy to follow.

In the diagram, we see a virtual disk that is configured to be stretched across locations, and protected by RAID-1 within each location. As a result, you will have two RAID-1 trees each with two components and a witness, and of course, you would have a witness component in the witness location. Now the question is, what happens when you place esxi-host-1 into maintenance mode? In this scenario, vSAN Enhanced Durability will want to create a “durability component”. This durability component is used to commit all new write IO to. This will allow vSAN to resync fast after maintenance mode, and enhances durability as we would still have 2 copies of the (new) data.

However, in the scenario above we only have 3 hosts per location. The question then is, where is this delta component created then? As normally with maintenance mode you would need a 4th host to move data to. Well, it is simple, in this case, what vSAN does is it creates a “durability component” on the host where the witness resides, within the location of course. Let me show you in a diagram, as that makes it clear instantly.

By adding the Durability component next to the witness on esxi-host-3, vSAN enhances durability even in this stretched cluster situation, as it provides a local additional copy of new writes. Now, of course I tested this in my lab. So for those who prefer to see a demo, check out the youtube video below.