Yellow Bricks

by Duncan Epping

6.2

Doing maintenance on a Two-Node (Direct Connect) vSAN configuration

· ·

I was talking to a partner and customer last week at a VMUG. They were running a two node (direct connect) vSAN configuration and had some issues during maintenance which were, to them, not easy to explain. What they did is they placed the host which was in the “preferred fault domain” in to maintenance mode. After they placed that host in to maintenance mode the link between the two hosts for whatever reason failed. After they rebooted the host in the preferred host it connected back to the witness but at this point in time the connection between the hosts had not returned yet. This confused vSAN and that resulted in the scenario where the VMs in the secondary fault domain were powered off. As you can imagine an undesired effect.

This issue is solved in the near future in a new version of vSAN, but for those who need to do maintenance on a two-node (direct connect) configuration (or a full site maintenance in a stretched environment) I would highly recommend the following simple procedure. This will need to be done when doing maintenance on the host which is in the “preferred fault domain”:

  • Change the preferred fault domain
    • Under vSAN, click Fault Domains and Stretched Cluster.
    • Select the secondary fault domain and click the Mark Fault Domain as preferred for Stretched Cluster icon
  • Place the host in to maintenance mode
  • Do your maintenance

Fairly straight forward, but important to remember…

Holiday gift: vSAN Essentials book available for free

· ·

Christmas is coming so Cormac and I figured we would do something special for everyone, after a long debate we decided to make the vSAN Essentials book available for free. Note that this is the “Essential Virtual SAN” book which was published by VMware Press / Pearson and is based on the 6.2 version of vSAN. The book however is still very relevant today, and of course we are considering doing an update of the content to either the latest release, or maybe even to an upcoming release. You can read the book online (which is what we recommend), but you can also download it as PDF, EPUB or MOBI format. Basically you can read it anywhere, anytime and using any device. Nice right!?!

We used the Gitbook platform to publish the book and decided to leverage the beta version of gitbook as it looks very clean and makes the content easy to read online. Also, I have used the gitbook platform in the past for the HA Deepdive, and I wanted to give back by beta testing their platform. Ah well, instead of rambling on, here’s the book:

vsan-essentials.com

If you find anything unusual, please leave a comment here. Hope you will enjoy it, and appreciates us (the authors) giving back to the community. If you do then I hope you will consider donating to charity, the amount doesn’t matter, all help is welcome! I personally support Hardcore Help Foundation, and I hope you will considering doing the same! A donation of 10€ will provide clean, safe water to a family for two years. They need your help to reach out to more families in need.

Which disk controller to use for vSAN

· ·

I have many customers going through the plan and design phase for implementing a vSAN based infrastructure. Many of them have conversations with OEMs and this typically results in a set of recommendations in terms of which hardware to purchase. One thing that seems to be a recurring theme is the question which disk controller a customer should buy. The typical recommendation seems to be the most beefy disk controller on the list. I wrote about this a while ago as well, and want to re-emphasize my thinking. Before I do, I understand why these recommendations are being made. Traditionally with local storage devices selecting the high-end disk controller made sense. It provided a lot of options you needed to have a decent performance and also availability of your data. With vSAN however this is not needed, this is all provided by our software layer.

When it comes to disk controllers my recommendation is simple: go for the simplest device on the list that has a good queue depth. Just to give an example, the Dell H730 disk controller is often recommended, but if you look at the vSAN Compatibility Guide then you will also see the HBA330. The big difference between these two is the RAID functionality offered on the H730 and the cache on the controller. Again, this functionality is not needed for vSAN, by going for the HBA330 you will save money. (For HP I would recommend the H240 disk controller.)

Having said that, I would at the same time recommend customers to consider NVMe for the caching tier instead of SAS or SATA connected flash. Why, well for the caching layer it makes sense to avoid the disk controller. Place the flash as close to the CPU as you can get for low latency high throughput. In other words, invest the money you are saving on the more expensive disk controller in NVMe connected flash for the caching layer.

List all “thick” swap files on vSAN

· ·

As some may know, on vSAN by default the swap file is a fully reserved file. This means that if you have a VM with 8GB of memory, vSAN will reserve 16GB capacity in total for it. 16GB? Yes, 16GB as the FTT=1 policy is also applied to it. In vSAN 6.2 we introduced the ability to have swap files created “thin” or “unreserved” I should probably say. You can simply do these by setting an advanced setting on each host in your cluster. (SwapThickProvisionDisabled) Now when you have set this and power-off/power-on your VMs the swap file is recreated and the swap file will be thin. Jase McCarty wrote a script that will set the setting for you in each host of your cluster, but the problem of course is how do you know which VM has the “new unreserved” swap file and which VM still has the fully reserved swap file. This is what a customer asked me last week.

I was sitting next to William at a session and I asked him this question. William went at it and knocked out a Python script which lists all VMs in a cluster which have a fully reserved swap file. Very useful for those who are moving to “unreserved / sparse” swap. This way you can figure out which VMs still need a reboot and reclaim that (unused) disk capacity.

Note, the “sparse” / “unreserved” swap files are only intended for environments which do not overcommit on memory. If you do overcommit on memory please ensure you have disk capacity available, as you will need the disk capacity as soon as the hypervisor wants to place memory pages in the swap file. If there’s no disk capacity available it will result in the VM failing.

Thanks William for knocking out this script so fast…

Can I front vSAN with a VAIO Caching Solution?

· ·

I had this question a couple of times already, so I figured I would write a quick post. In short: yes you can put a VAIO Filter in front of vSAN. The question really is, which one would you like to use and why?

First of all, the VAIO Filter needs to be certified to be placed in front of vSAN storage. Just like it needs to be certified for VMFS and NFS. You can go to the VAIO HCL and check this yourself:

In this example Infinio supports VVols, vSAN and VMFS. Great! Now the next question is: why? Well that is the bigger question, personally I don’t see a real compelling reason. For traditional storage it makes a lot of sense, as you want to keep IOs local and add cache in a “cheap way” instead of expanding, a potentially close to end of life, storage system. For vSAN that is different. vSAN has a distributed architecture and every node has a flash device that is being used for write caching, and also read caching in a hybrid scenario. If this is a new deployment invest your money in NVMe instead. If you want to repurpose existing hardware and it is not on the vSAN HCL, ask yourself if you should complicate your deployment or not?

I would personally recommend to keep it simple, but than again I can also understand you do not want to let flash resources go to waste if vSAN does not support the devices. If you want to go VAIO, make sure to check the HCL and take the potential risks and operational complexity in to account and weigh that against the cost.