TechPubs youtube videos

I just noticed these 3 cool TechPubs youtube videos, the techpubs channel has been around for a while and I have been enjoying their videos a lot. Recently a couple of new videos were released and I hadn’t gotten around to watching them yet, but these are definitely part of my favorites. One is on vSphere HA by the lead engineer: Keith Farkas (also a reviewer on our book), and two others are by Sachin Thakkar. Sachin is one of the leads on vSphere virtual networking features like VXLAN. I enjoyed watching these very much as they give a nice overview of what this feature is about in just a couple of minutes. I also personally feel it is nice to “get to know” the people behind this cool feature/technology…

Make sure to follow the TechPubs channel for more cool videos. Now it is back to christmas shopping again ;-)

vSphere HA


Renaming virtual machine files using SvMotion back in 5.0 U2

I have been pushing for this heavily internally together with Frank Denneman and it pleases me to say that it is finally back… You can rename your virtual machine files again using Storage vMotion as of 5.0 u2.

vSphere 5 Storage vMotion is unable to rename virtual machine files on completing migration
In vCenter Server , when you rename a virtual machine in the vSphere Client, the vmdk disks are not renamed following a successful Storage vMotion task. When you perform a Storage vMotion of the virtual machine to have its folder and associated files renamed to match the new name. The virtual machine folder name changes, but the virtual machine file names do not change.

This issue is resolved in this release


Those who want to know what else is fixed, you can find the full release notes here of both ESXi 5.0 U2 and vCenter 5.0 U2:

** do note that this fix is not part of 5.1 yet **

Using ESXTOP to check VAAI primitive stats

Yesterday a comment was made around a VAAI primitive on my article about virtual disk types and performance. In this case “write same” was mentioned and the comment was about how it would not be used when expanding a thin disk or lazy zero thick disk. Now the nice thing is that with ESXTOP you can actually see VAAI primitive stats. For instance “ATS” (locking) can be seen, but also… write same or “ZERO” as ESXTOP calls it.

If you open up ESXTOP and do the following you will see these VAAI primitive stats:

  • esxtop
  • press “u”
  • press “f”
  • press “o”
  • press “enter”

The screenshot below shows you what that should look like, nice right… In this case 732 blocks were zeroed out using the write-same / zero VAAI primitive.

VAAI primitive stats

Death to false myths: The type of virtual disk used determines your performance

I was reading Eric Sloof’s article last week about how the disk type will impact your performance. First of all, let me be clear that I am not trying to bash Eric here… I think Eric has a valid point, well at least in some cases and let me explain why.

On his quest of determining if the virtual disk type determines performance Eric tested the following on his environment:

  • Thin
  • Lazy zero thick
  • Eager zero thick

These are the three disk types you can choose from when creating a new virtual disk. The difference between them is simple. Thick are fully allocated virtual disk files. Lazy zero means that the disk is not zeroed out yet, eager zero means the full disk is zeroed out during the provisioning process. Thin, well I guess you know what it means… not fully allocated and not also not zeroed. This also implies that in the case of “thin” and “lazy zero thick” something needs to happen when a new “block” is accessed. This is what Eric showed in his test. But is that relevant?

First of all, the test was conducted with an Iomega PX6-300d. One thing to point out here  is that this device isn’t VAAI capable and limited from a performance perspective due to the CPU power and limited set of spindles. The lack of VAAI however impacts the outcome the most. This, in my opinion means, that the outcome cannot really be used by those who have VAAI capable arrays. The outcome would be different when one of the following two (or both) VAAI primitives are supported and used by the array:

  • Write Same aka “Zero Out”
  • ATS aka “Hardware Offloaded Locking”

Cormac Hogan wrote an excellent article on this topic and an excellent white paper, if you want to know more about ATS and Write Same make sure to read them.

Secondly, it is important to realize that most applications don’t have the same write pattern as the benchmarking tool used by Eric. In his situation the tool basically fills up an X amount of blocks sequentially to determine the max performance. Only if you do fill up a disk at once, or very large unwritten sections, you potentially could see a similar result. Let me emphasize that, could potentially.

I guess this myth was once a fact, back in 2009 a white paper was released about Thin/Thick disks. In this paper they demonstrate the difference between thin, lazy zero and eager zero thick… yes they do proof there is a difference but this was pre-VAAI. Now if you look at another example, a nice extreme example, which is a performance test done by my friends of Pure Storage you will notice there is absolutely no difference. This is an extreme example considering it’s an all-flash VAAI based storage system, nevertheless it proofs a point. But not just all-flash arrays see a huge improvement, take a look at this article by Derek Seaman about 3Par’s “write same” (zero’ing) implementation, I am pretty sure that in his environment he would also not see the huge discrepancy Eric witnessed.

I am not going to dispel this myth as it is a case of “it depends”. It depends on the type of array used, and for instance how VAAI was implemented as that could make difference. In most cases however it is safe to say that the performance difference will not be big, if noticeable at all during normal usage. I am not even discussing all the operational implications of using eager-zero-thick… (as Frank Denneman will respond soon to this blog post has a nice article about that. :-))

Death to false myths: Admission Control lowers consolidation ratio

Death to false myths probably sounds a bit euuhm well Dutch probably, or “direct” as others would label it. Lately I have seen some statements floating around which are either false or misused. One of them is around Admission Control and how it impacts consolidation ratio even if you are not using reservations. I have had multiple questions around this in the last couple of weeks and noticed this thread on VMTN.

The thread referred to is all about which Admission Control policy to use, as the selected policy potentially impacts the amount of virtual machines you can run on a cluster. Now lets take a look at the example in this VMTN thread, and I have rounded up some of the numbers to simplify things:

  • 7 host cluster
  • 512 GB of memory
  • 132 GHz of CPU resources
  • 217 MB of Memory Overhead (no reservations used)

So if you do the quick math. According to Admission Control (host failures example) you can power-on about ~2500 virtual machines. That is without taking N-1 resiliency in to account. When I take out the largest host we are still talking about ~1800 virtual machines that can be powered on. Yes that is 700 slots/virtual machines less due to the N-1, admission control needs to be able to guarantee that even if the largest host fails all virtual machines can be restarted.

Considering we have 512GB in total that means that if those 1800 virtual machines on average actively use 280MB we will see TPS / swapping / ballooning / compression. (512GB / 1800 VMs) Clearly you want to avoid most of these, swapping / ballooning / compression that is. Especially considering most VMs are typically provisioned with 2GB of memory or more.

So what does that mean or did we learn? Two things:

  • Admission Control is about guaranteeing virtual machine restarts
  • If you set no reservation you can power-on an insane amount of virtual machines

Let me reemphasize the last bullet, you can power-on an INSANE amount of virtual machines on just a couple of hosts when no reservations are used. In this case HA would allow for 1800 virtual machines to be powered-on before it starts screaming it is out of resources. Is that going to work in real life, would your virtual machines be happy with the amount of resources they are getting? I don’t think so… I don’t believe that 280MB of physically backed memory is sufficient for most workloads. Yes, maybe TPS can help a bit, but chances of hitting the swap file are substantial.

Let it be clear, admission control is no resource management solution. It is only guaranteeing virtual machines can be restarted and if you have no reservations set then the numbers you will see are probably not realistic. At least not from a user experience perspective. I bet your users / customers would like to have a bit more resources available than just the bare minimum required to power-on a virtual machine! So don’t let these numbers fool you.