Yellow Bricks

by Duncan Epping

Storage

Does a vSAN IO Limit impact resync traffic?

Leave a Comment

A question just came in, and I figured other people may have the same question so I would share it. The question was if a vSAN IO limit would impact resync traffic or for instance SvMotion? In this case the customer defines limits within each policy to ensure VMs do not interfere with other VMs or excessively uses IO resources. Especially in cloud environments this can be useful, or when running production and test/dev on the same cluster. The concern, of course, was if this limit would impact for instance recovery times after a failure. Because you can imagine that a limit of 50 IOPS would be devastating when a VM (or multiple VMs) need to have objects resynced.

The answer is simple: no, the IO limit specified within a policy does not impact resync traffic (or SvMotion for that matter). It only applies to Guest IO to a VMDK, namespace or swap object. Which means that it is safe to set limits when it comes to recovery times.

Major vSAN Milestone: 20K customers – Celebrating by dropping the price of our book with 50%!

2 Comments

I haven’t done one of these in a while, and as it is a question that comes up regularly during customer conversations I figured I would share a nice quote from the VMware earnings call. But before I do I want to thank every VMware employee, partner and customer who helped us reaching this major milestone. Sometimes customers ask how invested VMware is in storage, well very invested. Determined to remain the number 1 player in the hyperconverged and hybrid cloud world, and the below numbers show why!

vSAN license bookings grew over 50% year-over-year in Q1 with a total customer count growing to over 20,000. (seekingalpha.com)

Yes, that is 20,000 customers indeed. Actually, more than 20k customers. Which, again, is a great success and would not have been possible without the help from you guys. So to thank all of you Cormac and I have decided to lower the price of our book temporarily. For 1 week, today until Friday the 7th, we have lowered the price of the book by ~50%. This means that on the Amazon US store the book will be 20 USD for the paper version, and only 5 USD for the ebook. So pick it up! (It may take a day for the price change to reach some of the Amazon stores…)

<UPDATE>

It seems the price has been pushed down to all “local” amazon websites. So go to your local website and pick up the book for 50% of the previous price. Links to most localized websites here:

Impact of adding Persistent Memory / Optane Memory devices to your VM

Leave a Comment

I had some questions around this in the past month, so I figured I would share some details around this. As persistent memory (Intel Optane Memory devices for instance) is getting more affordable and readily available more and more customers are looking to use it. Some are already using it for very specific use cases, usually in situations where the OS and the App actually understand the type of device being presented. What does that mean? At VMworld 2018 there was a great session on this topic and I captured the session in a post. Let me copy/paste the important bit for you, which discusses the different modes in which a Persistent Memory device can be presented to a VM.

  • vPMEMDisk = exposed to guest as a regular SCSI/NVMe device, VMDKs are stored on PMEM Datastore
  • vPMEM = Exposes the NVDIMM device in a “passthrough manner, guest can use it as block device or byte addressable direct access device (DAX), this is the fastest mode and most modern OS’s support this
  • vPMEM-aware = This is similar to the mode above, but the difference is that the application understands how to take advantage of vPMEM

But what is the problem with this? What is the impact? Well when you expose a Persistent Memory device to the VM, it is not currently protected by vSphere HA, even though HA may be enabled on your cluster. Say what? Yes indeed, the VM which has the PMEM device presented to it will be disabled for vSphere HA! I had to dig deep to find this documented anywhere, and it is documented in this paper. (Page 47, at the bottom.) So what works and what not? Well if I understand it correctly:

  • vSphere HA >> Not supported on vPMEM enabled VMs, regardless of the mode
  • vSphere DRS >> Does not consider vPMEM enabled VMs, regardless of the mode
  • Migration of VM with vPMEM / vPMEM-aware >> Only possible when migrating to host which has PMEM
  • Migration of VM with vPMEMDISK >> Possible to a host without PMEM

Also note, as a result (data is not replicated/mirrored) a failure could potentially lead to loss of data. Although Persistent Memory is a great mechanism to increase performance, it is something that should be taken into consideration when you are thinking about introducing it into your environment.

Oh, if you are wondering why people are taking these risks in terms of availability, Niels Hagoort just posted a blog with a pointer to a new PMEM Perf paper which is worth reading.

 

 

Mixing versions of ESXi in the same vSphere / vSAN cluster?

2 Comments

I have seen this question being asked a couple of times in the past months, and to be honest I was a bit surprised people asked about this. Various customers were wondering if it is supported to mix versions of ESXi in the same vSphere or vSAN Cluster? I can be short whether this is support or not, yes it is. Would I recommend it? No, I would not!

Why not? Well mainly for operational reasons, it just makes life more complex. Just think about a troubleshooting scenario, you now need to remember which version you are running on which host and understand the “known issues” for each version. Also, for vSAN things are even more complex as you could have “components” running on a different version of ESXi. On top of that, it could even be the case that a certain command or esxcli namespace is not available on a particular version of ESXi.

Another concern is when doing upgrades or updates, you need to take the current version into account when updating, or more importantly when upgrading! Also, remember that firmware/driver combination may be different for a particular version of vSphere/vSAN as well, this could also make life more complex and definitely increases the chances of making mistakes!

Is this documented anywhere? Well, there are two KBs which hint at this, for vSAN specifically I have requested our team to document this in an upcoming document. The KBs that hint at this are below.

vSAN Stretched Cluster failure scenarios and component votes

1 Comment

I was at a customer last week and had an interesting question about the vSAN voting mechanism. This customer had a stretched cluster and used RAID-5 within each location to protect the data on top of replicating across locations. During certain failure scenarios unexpectedly the data remained available, of course, it is great that you have higher availability than expected, but why did this happen? What this customer tested was powering off the Witness (which is deemed as a site failure) and next powered of 2 hosts in 1 location, which exceeds the “failures to tolerate” in a single location. You would expect, based on all documentation so far, that the data would be unavailable. Well for some VMs this was the case, but for others, that was not the case. Why is this? Well, it is all about the vote count in this case. Look at the below diagram and the number of votes for each component first.

In the above scenario if the Witness (W) fails we have 4 votes less. Out of a total of 13 that is not a problem. If two additional hosts fail, this is most likely still not a problem, even though you are exceeding the provided “failures to tolerate”. However, if by any chance Host1 is one of those failed hosts then you would lose quorum. Host1 has a component with 2 votes. So if host1 has failed and the witness has failed and host2 for instance, you have now lost 7 out of 13 votes. This means quorum is lost. Please note that that single component with 2 votes is random. For a different VM/Object it could be that the component which is placed on host6 or host7 has 2 votes.

Another thing to point out, if host5-8 all would fail the data is still available. However, if then host3 and host4 would fail the object would become unavailable. Even though you still would have quorum across locations, you have now also exceeded the specified “failures to tolerate” within the location. This is also something that will be taken in to account.

I hope that helps.