Yellow Bricks

by Duncan Epping

5.0

vSphere 5.0 vMotion Enhancements

· ·

**disclaimer: this article is an out-take of our book: vSphere 5 Clustering Technical Deepdive**

There are some fundamental changes when it comes to vMotion scalability and performance in vSphere 5.0. Most of these changes have one common goal: being able to vMotion ANY type of workload. It doesn’t matter if you have a virtual machine with 32GB of memory that is rapidly changing memory pages any more with the the following enhancements:

  • Multi-NIC vMotion support
  • Stun During Page Send (SDPS)

Multi-NIC vMotion Support

One of the most substantial and visible changes is multi-NIC vMotion capabilities. vMotion is now capable of using multiple NICs concurrently to decrease the amount of time a vMotion takes. That means that even a single vMotion can leverage all of the configured vMotion NICs. Prior vSphere 5.0, only a single NIC was used for a vMotion enabled VMkernel. Enabling multiple NICs for your vMotion enabled VMkernel’s will remove some of the constraints from a bandwidth/throughput perspective that are associated with large and memory active virtual machines. The following list shows the currently supported maximum number of NICs for multi-NIC vMotion:

  • 1GbE – 16 NICs supported
  • 10GbE – 4 NICs supported

It is important to realize that in the case of 10GbE interfaces, it is only possible to use the full bandwidth when the server is equipped with the latest PCI Express busses. Ensure that your server hardware is capable of taking full advantage of these capabilities when this is a requirement.

Stun During Page Send

A couple of months back I described this cool vSphere 4.1 vMotion enhancement called Quick Resume and now it is replaced with Stun During Page Send, or also often referred to as “Slowdown During Page Send” is a feature that “slowsd own” the vCPU of the virtual machine that is being vMotioned. Simply said, vMotion will track the rate at which the guest pages are changed, or as the engineers prefer to call it, “dirtied”. The rate at which this occurs is compared to the vMotion transmission rate. If the rate at which the pages are dirtied exceeds the transmission rate, the source vCPUs will be placed in a sleep state to decrease the rate at which pages are dirtied and to allow the vMotion process to complete. It is good to know that the vCPUs will only be put to sleep for a few milliseconds at a time at most. SDPS injects frequent, tiny sleeps, disrupting the virtual machine’s workload just enough to guarantee vMotion can keep up with the memory page change rate to allow for a successful and non-disruptive completion of the process. You could say that, thanks to SDPS, you can vMotion any type of workload regardless of how aggressive it is.

It is important to realize that SDPS only slows down a virtual machine in the cases where the memory page change rate would have previously caused a vMotion to fail.

This technology is also what enables the increase in accepted latency for long distance vMotion. Pre-vSphere 5.0, the maximum supported latency for vMotion was 5ms. As you can imagine, this restricted many customers from enabling cross-site clusters. As of vSphere 5.0, the maximum supported latency has been doubled to 10ms for environments using Enterprise Plus. This should allow more customers to enable DRS between sites when all the required infrastructure components are available like, for instance, shared storage.

vSphere 5.0: vMotion enhancement, tiny but very welcome!

· ·

vSphere 5.0 has many new compelling features and enhancements. Sometimes though it is that little tiny enhancement that makes life easier. In this case I am talking about a tiny enhancements for vMotion which I know many of you will appreciate. It is something that both Frank Denneman and I have addressed multiple times with our engineers and finally made it into this release.

Selection Resource Pools?

I guess we have all cursed when we had to manually migrate VMs around and accidentally selected the wrong Resource Pool. This operational “problem” has finally been resolved and I am very happy about it. As of 5.0 the “source” resource pool will automatically be selected. Of course it is still possible to override this and to select a different resource pool but in most cases “next – next – finish” will be just fine.

ESXi 5.0 and Scripted Installs

· ·

When I was playing with ESXi 5.0 in my lab I noticed some changes during the installation process. Of course I had not bothered to read the documentation but when I watched the installer fail I figured it might make sense to start reading. I’ve documented the scripted installation procedure multiple times by now.

With ESXi 5.0 this has been simplified, this is what it looks like today:

I also want to point out that many of the standard installation commands have been replaced, removed or are not supported anymore. I created a simple script to automatically install an ESXi 5.0 host. It creates a second vSwitch and a second VMkernel for vMotion. It enables both the local and remote TSM and sets the default PSP for the EMC VMAX to Round Robin. As you can see there is a huge shift in this script towards esxcli. Although some of the old “esxcfg-*” commands might still be working they are deprecated and no longer supported. The new standard is esxcli, make sure you get familiarized with it and start using it today as over time this will be the only CLI tool available.


# Sample scripted installation file
# Accept the VMware End User License Agreement
vmaccepteula
# Set the root password for the DCUI and ESXi Shell
rootpw mypassword
# Install on the first local disk available on machine
install --firstdisk --overwritevmfs
# Set the network to DHCP on the first network adapater, use the specified hostname and do not create a portgroup for the VMs
network --bootproto=dhcp --device=vmnic0 --addvmportgroup=0
# reboots the host after the scripted installation is completed
reboot

 

%firstboot --interpreter=busybox
# Add an extra nic to vSwitch0 (vmnic2)
esxcli network vswitch standard uplink add --uplink-name=vmnic2 --vswitch-name=vSwitch0
#Assign an IP-Address to the first VMkernel, this will be used for management
esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=192.168.1.41 --netmask=255.255.255.0 --type=static
# Add vMotion Portgroup to vSwitch0, assign it VLAN ID 5 and create a VMkernel interface
esxcli network vswitch standard portgroup add --portgroup-name=vMotion --vswitch-name=vSwitch0
esxcli network vswitch standard portgroup set --portgroup-name=vMotion --vlan-id=5
esxcli network ip interface add --interface-name=vmk1 --portgroup-name=vMotion
esxcli network ip interface ipv4 set --interface-name=vmk1 --ipv4=192.168.2.41 --netmask=255.255.255.0 --type=static
# Enable vMotion on the newly created VMkernel vmk1
vim-cmd hostsvc/vmotion/vnic_set vmk1
# Add new vSwitch for VM traffic, assign uplinks, create a portgroup and assign a VLAN ID
esxcli network vswitch standard add --vswitch-name=vSwitch1
esxcli network vswitch standard uplink add --uplink-name=vmnic1 --vswitch-name=vSwitch1
esxcli network vswitch standard uplink add --uplink-name=vmnic3 --vswitch-name=vSwitch1
esxcli network vswitch standard portgroup add --portgroup-name=Production --vswitch-name=vSwitch1
esxcli network vswitch standard portgroup set --portgroup-name=Production --vlan-id=10
# Set DNS and hostname
esxcli system hostname set --fqdn=esxi5.localdomain
esxcli network ip dns search add --domain=localdomain
esxcli network ip dns server add --server=192.168.1.11
esxcli network ip dns server add --server=192.168.1.12
# Set the default PSP for EMC V-MAX to Round Robin as that is our preferred load balancing mechanism
esxcli storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_SYMM
# Enable SSH and the ESXi Shell
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

For more deepdive information read William’s post on ESXCLI and Scripted Installs.

Paperback editions finally available of vSphere 5 Clustering Tech Deepdive

· ·

I guess Amazon and Createspace had enough of getting those phone calls three or four times from Frank every day since Tuesday as magically the book popped up on Amazon overnight. It took a while and our apologies for that, here are the links to all three books:

Amazon:
eBook (Kindle) – $ 9.99
Black & White Paperback – $ 29.95
Full Color Paperback – $ 49.95

I hope you guys will enjoy it. Frank and I are considering to do a book signing at VMworld, let me know if that is something you would be interested in!

vSphere 5.0: Storage initiatives

· ·

Storage has been my primary focus for the 5.0 launch. The question often asked when talking about the separate components is how it all fits together. Lets first list some of the new or enhanced features:

  • VMFS-5
  • vSphere Storage APIs – Array Integration aka VAAI
  • vSphere Storage APIs – Storage Awareness aka VASA
  • Profile-Driven Storage (VM Storage Profiles in the GUI)
  • Storage I/O Control
  • Storage DRS

I wrote separate articles about all of these features and hopefully you have read them and already see the big picture. If you don’t than this is a good opportunity to read them or head over to the vSphere Storage Blog for more details on some of these.. I guess the best way to explain it is by using an example of what life could be like when using all of these new or enhanced features compared to what is used to be like:

The Old Way: Mr Admin is managing a large environment and currently has 300 LUNs each being 500GB divided across three 8 hosts clusters. He is maintaining a massive spreadsheet with storage characteristics and runs scripts to validate virtual machines are place on the correct tier of storage. He is leveraging SIOC to avoid the noisy neighbor problem and leveraging the VAAI primitives to offload some of the tasks to the array. Still he spends a lot of time waiting, monitoring, managing virtual machines and datastores.

vSphere 5.0: Mr Admin is managing a large environment and currently has 60 thin provisioned 2.5TB LUNs presented to a single cluster. Mr Admin defines several storage tiers using VM Storage Profiles detailing storage characteristics provided through VASA. Per tier based on the information provided through VASA a Datastore Cluster is created. Datastore Clusters form the basis of Storage DRS and Storage DRS will be responsible for initial placement and preventing both IO and diskspace bottlenecks in your environment. As Storage IO Control is automatically enabled when SDRS IO balancing is enabled the noisy neighbor problem will also be eliminated. When provisioning a new virtual machine Mr Admin simple picks the appropriate VM Storage Profile and selects the compliant Datastore Cluster. If in any case Storage DRS would move things around, the “Reclaim Dead Space” feature of VAAI is used to unmap the blocks from the source datastore so that these can be re-used if and when needed.

No more spreadsheets, no extensively monitoring diskspace / latency, no more manual validation of virtual machine placement… It is all about ease of management, reducing operational effort and offloading tasks to vCenter or even your storage array!