Stretched vCloud Director infrastructure

A while back I wrote about design considerations when designing or building a stretched vCloud Director infrastructure. Since then I have been working on a document in collaboration with Lee Dilworth, and this document should be out soon hopefully. As various people have asked for the document I decided to throw it in to this blog post so that the details are already out there.

** Disclaimer: this article has not been reviewed by the technical marketing team yet, this is a preview of what will possibly be published. When the official document is published I will add a link to this article **

Introduction

VMware vCloud® Director™ 5.1 (vCloud Director) gives enterprise organizations the ability to build secure private clouds that dramatically increase datacenter efficiency and business agility. Coupled with VMware vSphere® (vSphere), vCloud Director delivers cloud computing for existing datacenters by pooling vSphere virtual resources and delivering them to users as catalog-based services. vCloud Director helps you build agile infrastructure-as-a-service (IaaS) cloud environments that greatly accelerate the time-to-market for applications and responsiveness of IT organizations.

Resiliency is a key aspect of any infrastructure but is even more important in “Infrastructure as a Service” (IaaS) solutions. This solution overview was developed to provide additional insight and information in how to architect and implement a vCloud Director based solution on a vSphere Metro Storage Cluster infrastructure.

Architecture Introduction

This architecture consists of two major components. The first component is the geographically separated vSphere infrastructure based on stretched storage solution, here after referred to as the vSphere Metro Storage Cluster (vMSC) infrastructure. The second component is vCloud Director.

Note –  Before we dive in to the details of the solution we would like to call out the fact that vCloud Director is not site aware. If incorrectly configured availability could be negatively impacted in certain failure scenarios.

[Read more...]

My Bookstore

I had various people asking why I didn’t make it easier to buy my books via my website. Greg Schulz from StorageIOblog.com mentioned that he created an amazon store. So I took a peek at how he set it up and kinda did something similar. If you click “My Bookstore” at the top you will be automatically brought to a custom Amazon store. It will list all my books on the first page, paper copies that is. On the right side you have two more options:

  • Kindle Items – ebook versions of what I have published so far
  • Books by Friends – exactly what it says… books that my friends have written

Over time of course new books will be added to the appropriate sections. Thanks and I hope it makes things easier.

Faking an SSD in your virtualized vSphere lab

I have written about this before (and so has William Lam, so all credits go to William), but I wanted to note down these commands for my own use as I find myself digging around often for the same commands these days. So what is my goal: Faking an SSD in my virtualized vSphere lab.

In my lab I have a bunch of virtualized ESXi hosts. Those hosts have multiple disks and I want to mark one of those disks as SSD. To keep things simple I set things up as follows. Just to point out, I use 0:0 / 1:0 / 2:0 so that each device gets a new controller and is easy to identifiy:

  • First Disk – ESXi install disk – 5GB – SCSI 0:0
  • Second Disk – Fake SSD – 40GB – SCSI 1:0
  • Third Disk – Large disk – 1TB – SCSI 2:0

When I boot all disks are recognized as regular disks and in some cases as non-local. In my testing I need local disks and need SSD. So this is what I did to get exactly that. With the first command I mark the “second disk” as SSD and local. With the second command I mark the third disk as local. Next I reclaim the devices so that the new SATP rules are applied.

esxcli storage nmp satp rule add --satp VMW_SATP_LOCAL --device mpx.vmhba2:C0:T0:L0 --option "enable_local enable_ssd"
esxcli storage nmp satp rule add --satp VMW_SATP_LOCAL --device mpx.vmhba3:C0:T0:L0 --option "enable_local"
esxcli storage core claiming reclaim -d mpx.vmhba2:C0:T0:L0
esxcli storage core claiming reclaim -d mpx.vmhba3:C0:T0:L0

Next you can simply validate if it has worked by typing the following for device vmhba2 and 3 (if you replace the 2 with a 3 ofcourse) :

esxcli storage core device list --device=mpx.vmhba2:C0:T0:L0

As you can see, faking an SSD is fairly straight forward. Note that even if you have an SSD drive you still might need to do this. In some cases the SSD drive is not recognized and you will need to create a rule for it manually.

Percentage Based Admission Control gives lower VM restart guarantee?

Those who have configured vSphere HA have all seen that section where it asks if you want to use admission control or not. Of course if you decide you want to use it, and you should want this, then the next question that comes is which one do you want to use? I have always preferred the “Percentage Based Admission Control” policy. For some reason though there are people who think that the percentage based admission control policy rules out large VMs from being restarted or offers a lower guarantee.

The main perception that people have is that the percentages based admission control policy gives lower guarantees of virtual machines being restarted than the “host failures” admission control policy. So let break it down, and I mean BREAK IT DOWN, by using an example.

Example

  • 5 hosts
  • 200GB of Memory in cluster
  • 20GHz of CPU in cluster

If no reservations are set:

Percentage Based will do the following:

  1. The Percentage Based policy will take the total amount of resources and subtract the amount of resources reserved for fail-over. If that percentage is for instance 20% than 40GB and 4GHz are subtracted. Which means 160GB and 16GHz are left.
  2. The reserved resources for every virtual machine that is powered on is subtracted from what the outcome of 1. was. If no reservation is set memory then memory overhead is subtracted, if the memory overhead is 200MB then 200MB is subtracted from the 160GB that was left resulting in 159,8GB being available. For CPU the default of 32MHz will be used.
  3. You can power-on virtual machines until the amount of available resources, according to HA Admission Control, is depleted, yes many VMs in this case.

Host Failures will do the following:

  1. The Host Failures policy will calculate the amount of slots. A slot is formed out of two components: memory and cpu. As no reservation is used the default for CPU is used which is 32MHz, with vSphere 5.0 and higher. For memory the largest memory overhead size is used, in this scenario there could be a variety of sizes lets say the smallest is 64MB and the largest 300MB. Now 300MB will be used for the Memory Slot size.
  2. Now that the slotsize is known Admission Control will look for the host with the most slots (available resources / slot size) and subtract those slots from the total amount of available slots. (If one host failure is specified). Every time a VM is started a slot is subtracted. If a VM is started with a higher memory reservation we go back to 1 and the math will need to be done again.
  3. You can power-on virtual machines until you are out of slots, again… many VMs.

If reservations are set:

Percentage Based will do the following:

  1. The Percentage Based policy will take the total amount of resources and subtract the amount of resources reserved for fail-over. If that percentage is for instance 20% than 40GB and 4GHz are subtracted. Which means 160GB and 16GHz are left.
  2. The reserved resources for every virtual machine that is powered on is subtracted from what the outcome of 1 was. So if 10GB of memory was reserved, then 10GB is subtracted resulting in 150GB being available.
  3. You can power-on virtual machines until available resources are depleted (according to HA Admission Control), but as reservations are used you are “limited” in terms of the amount of VMs you can power-on.

Host Failures will do the following:

  1. The Host Failures policy will calculate the amount of slots. A slot is formed out of two components: memory and cpu. As a reservation is used for memory but not for CPU the default for CPU is used which is 32MHz, with vSphere 5.0 and higher. For memory there is a 10GB reservation set. 10GB will be used for the Memory Slot size.
  2. Now that the slotsize is known Admission Control will look for the host with the most slots (available resources / slot size) and subtract those slots from the total amount of available slots. (If one host failure is specified). Every time a VM is started a slot is subtracted, yes that is a 10GB memory slot, even if it has for instance a 2GB reservation. If a VM is started with a higher memory reservation we go back to 1 and the math will need to be done again.
  3. You can power-on virtual machines until you are out of slots, as a high reservation is set you will be severely limited!

Now you can imagine that “Host Failures” can be on the safe side… If you have 1 reservation set the math will be done with that reservation. This means that a single 10GB reservation will impact how many VMs you can power-on until HA screams that it is out of resources. But at least you are guaranteed you can power them on right? Well yes, but realistically speaking people disable Admission Control at this point as that single 10GB reservation allows you to power on just a couple of VMs. (16 to be precise.)

But but that beats Percentage Based right… because if I have a lot of VMs who says my VM with 10GB reservation can be restarted? First of all, if there are no “unreserved resources” available on any given host to start this virtual machine then vSphere HA will ask vSphere DRS to defragment the cluster.As HA Admission Control had already accepted this virtual machine to begin with, chances are fairly high that DRS can solve the fragmentation.

Also, as the percentage based admission control policy uses reservations AND memory overhead… how many virtual machines do you need to have powered-on before your VM with 10 GB memory reservation is denied to be powered-on? It would mean that none of the hosts has 10GB of unreserved memory available. That is not very likely as that means you would need to power-on hundreds of VMs… Probably way too many for your environment to ever perform properly. So chances of hitting this scenario are limited, extremely small.

Conclusion

Although theoretically possible, it is very unlikely you will end up in situation where one or multiple virtual machines can not be restarted when using the Percentage Based Admission Control policy. Even if you are using reservations on all virtual machines then this is unlikely as the virtual machines have been accepted at some point by HA Admission Control and HA will leverage DRS to defragment resources at that point. Also keep in mind that when using reservations on all virtual machines that Host Failures is not an option as it will skew your numbers as it does the math with “worst case scenario”, a single 10GB reservation can kill your ROI/TCO.

In short: Go Percentage Based!

Automating ESXi host level changes without opening SSH

I have been asked by many if it is possible automating ESXi host level changes without opening SSH. In many organizations people are prohibited to open SSH however they do have the need to make certain changes on a host level. One of those changes for instance is in a stretched cluster environment where “disk.terminateVMOnPDLDefault” needs to be set to true. This setting can only be configured in /etc/vmware/settings unfortunately. So how do you automate this?

Andreas Peetz from V-Front.de came up with an awesome solution. He created a plugin to esxcli allowing you to run commands on an ESXi host. So in other words, when you install his plugin (it is a vib) you can remotely fire off a command on an ESXi host as if you are sitting behind that host.

How does that work? Well first of all you install the vib Andreas created. (Or include it in your image.) When it is installed you can simply run the following on any machine that has the vSphere CLI installed:

esxcli -s hostname -u username -p password shell cmd -c "command"

Awesome right?! I think so, this is probably one of the coolest things I have seen in a while. Very clever solution, once again… awesome work Andreas and head over to V-Front.de to get more details and the actually download of this plugin!

** Disclaimer: implementing this solution could result in an unsupported configuration. This article was published to demonstrate the capabilities of esxcli and for educational purposes **

Upgrade vCloud Director 1.5 on vSphere 5.1 to vCD 5.1.1?

One of my colleagues, Matthew Meyer, posted a list of cool videos he produced. These videos show how to upgrade vCloud Director 1.5 on vSphere 5.0 to vCloud Director 5.1.1 running on vSphere 5.1.  Nice right?! Thanks Matt for creating these, awesome work. (I normally don’t use the “read more” option, but as there are 8 videos in total I will only show two on the front page. Hit “Continue Reading” if you want to see the rest!)

VMware vCenter Server 5.0 to 5.1 Upgrade

VMware vCenter Single Sign-On Installation

[Read more...]