Latest Revision:
Dec 9, 2010

Download:
http://www.vmware.com/files/pdf/techpaper/VMW-Server-WP-BestPractices.pdf

Description

This paper describes best practices and guidance for properly deploying VMware HA in VMware vSphere 4.1.  These include discussions on proper network and storage design, and recommendations on settings for host isolation response and admission control.

Storage IO Control Best Practices:

• Enable Storage IO Control on all datastores
• Avoid external access for SIOC enabled datastores
  • To avoid any interference SIOC will stop throttling, more info here.
• When multiple datastores share the same set of spindles ensure all have SIOC enabled with comparable settings and all have sioc enabled.
• Change latency threshold based on used storage media type:
  • For FC storage the recommended latency threshold is  20 – 30 MS
  • For SAS storage the recommended latency threshold is  20 – 30 MS
  • For SATA storage the recommended latency threshold is 30 – 50 MS
  • For SSD storage the recommended latency threshold is 15 – 20 MS
• Define a limit per VM for IOPS to avoid a single VM flooding the array
  • For instance limit the amount of IOPS per VM to a 1000

Without setting the shares you can simply enable Storage I/O controls on each datastore. This will prevent any one VM from monopolizing the datatore by leveling out all requests for I/O that the datastore receives.

This is exactly the reason why I would recommend anyone who has a large environment, and even more specifically in cloud environments, to enable SIOC. Especially in very large environments where compute, storage and network resources are designed to accommodate the highest common factor it is important to ensure that all entities can claim their fair share of resource and in this case SIOC will do just that.

Now the question is how does this actually work? I already wrote a short article on it a while back but I guess it can’t hurt to reiterate thing and to expand a bit.

First a bunch of facts I wanted to make sure were documented:

• SIOC is disabled by default
• SIOC needs to be enabled on a per Datastore level
• SIOC only engages when a specific level of latency has been reached
• SIOC has a default latency threshold of 30MS
• SIOC uses an average latency across hosts
• SIOC uses disk shares to assign I/O queue slots
• SIOC does not use vCenter, except for enabling the feature

When SIOC is enabled disk shares are used to give each VM its fair share of resources in times of contention. Contention in this case is measured in latency. As stated above when latency is equal or higher than 30MS, and the statistics around this are computed every 4 seconds, the “datastore-wide disk scheduler” will determine which action to take to reduce the overall / average latency and increase fairness. I guess the best way to explain what happens is by using an example.

As stated earlier, I want to keep this post fairly simple and I am using the example of an environment where every VM will have the same amount of shared. I have also limited the amount of VMs and hosts in the diagrams. Those of you who attended VMworld session TA8233 (Ajay and Chethan) will recognize these diagrams, I recreated and slightly modified them.

The first diagram shows three virtual machines. VM001 and VM002 are hosted on ESX01 and VM003 is hosted on ESX02. Each VM has disk shares set to a value of 1000. As Storage I/O Control is disabled there is no mechanism to regulate the I/O on a datastore level. As shown in the bottom by the Storage Array Queue in this case VM003 ends up getting more resources than VM001 and VM002 while all of them from a shares perspective were entitled to the exact same amount of resources. Please note that both Device Queue Depth’s are 32, which is the key to Storage I/O Control but I will explain that after the next diagram.

As stated without SIOC there is nothing that regulates the I/O on a datastore level. The next diagram shows the same scenario but with SIOC enabled.

After SIOC has been enabled it will start monitoring the datastore. If the specified latency threshold has been reached (Default: Average I/O Latency of 30MS) for the datastore SIOC will be triggered to take action and to resolve this possible imbalance. SIOC will then limit the amount of I/Os a host can issue. It does this by throttling the host device queue which is shown in the diagram and labeled as “Device Queue Depth”. As can be seen the queue depth of ESX02 is decreased to 16. Note that SIOC will not go below a device queue depth of 4.

Before it will limit the host it will of course need to know what to limit it to. The “datastore-wide disk scheduler” will sum up the disk shares for each of the VMDKs. In the case of ESX01 that is 2000 and in the case of ESX02 it is 1000. Next the  ”datastore-wide disk scheduler” will calculate the I/O slot entitlement based on the the host level shares and it will throttle the queue. Now I can hear you think what about the VM will it be throttled at all? Well the VM is controlled by the Host Local Scheduler (also sometimes referred to as SFQ), and resources on a per VM level will be divided by the the Host Local Scheduler based on the VM level shares.

I guess to conclude all there is left to say is: Enable SIOC and benefit from its fairness mechanism…. You can’t afford a single VM flooding your array. SIOC is the foundation of your (virtual) storage architecture, use it!

ref:
PARDA whitepaper
storage i/o control whitepaper
vmworld storage drs session
vmworld storage i/o control session

COS Memory:

• Although COS memory is “dynamic” I still always increase it to the full 800. The overhead of this in most of the servers(usually always 48GB+) is tiny. (-)

Host Configuration:

• Hostnames in lowercase characters; to avoid any HA issues. (*)
• I never change the name of the Service console portgroup, people are used to this name changing it leads to confusion in most cases and it is a critical part of your host. (-)
• Avoid using agents within the Service Console. (*)

vSwitch settings:

• Mac address changes: Reject (-)
  A best practice recommended by VMware PSO to ensure that when someone changes a MAC within the OS all inbound packets are dropped.
• Forged Transmit: Reject (-)
  Setting Forged Transmits to reject ensures that the originator of the packet is validated. Any outbound frame with a MAC address that is different from the one currently set on the adapter will be dropped. Again a best practice recommended by VMware PSO.

Single initiator zoning is something we have always recommend in the field (VMware PSO Consultants/Architects) and something that is clearly mentioned in our documentation… at least that’s what I thought.

On page 31 of the SAN Design and Deploy guide we clearly state the following:

When a SAN is configured using zoning, the devices outside a zone are not visible to the devices inside the zone. When there is one HBA or initiator to a single storage processor port or target zone, it is commonly referred to as single zone. This type of single zoning protects devices within a zone from fabric notifications, such as Registered State Change Notification (RSCN) changes from other zones. In addition, SAN traffic within each zone is isolated from the other zones. Thus, using single zone is a common industry practice.

That’s crystal clear isn’t it? Unfortunately there’s another document floating around which is called “Fibre Channel SAN Configuration Guide” and this document states the following on page 36:

• ESX Server hosts that use shared storage for virtual machine failover or load balancing must be in one zone.
• If you have a very large deployment, you might need to create separate zones for different areas of functionality. For example, you can separate accounting from human resources.

So which one is correct and which one isn’t? I don’t want any confusion around this. The first document, the SAN Design and Deploy guide is correct. VMware recommends single initiator zoning. Of course if you want to do “single initiator / single target” that would even be better, but single initiator is the bare minimum. Now let’s hope the VMware Tech Writers can get that document fixed…

There’s a simple formula that can be used to calculate how much disk space the metadata will consume. This formula used to be part of the “SAN System Design and Deployment Guide” (January 2008) but seems to have been removed in the updated versions.

Approximate metadata size in MB = 500MB + ((LUN Size in GB – 1) x 0.016KB)

For a 500GB LUN this would result in the following:

500 MB + ((500 - 1) x 0.016KB) = 507,984 MB
Roughly 1% of the total disk size used for metadata

For a 1500MB LUN this would result in the following:

500 MB + ((1.5 - 1) x 0.016KB) = 500,008 MB
Roughly 33% of the total disk size used for metadata

As you can see for a large VMFS volume(500GB) the disk space taken up by the metadata is only 1% and can almost be neglected, but for a very small LUN it will consume a lot of the disk space and needs to be taken into account….

[UPDATE]: As mentioned in the comments, the formula seems to be incorrect. I’ve looked into it and it appears that this is the reason it was removed from the documentation. The current limit for metadata is 1200MB and this should be the number you should use for sizing your datastores.

I think the key take aways are:

• Block sizes do not impact performance, neither large or small, as the OS dictates the block sizes used.
• Large block sizes do not increase storage overhead as sub-blocks are used for small files. The sub-blocks are always 64KB.
• With thin provisioning there theoretically are more locks when a thin disk is growing but the locking mechanism has been vastly improved with vSphere which means this can be neglected. A thin provisioned VMDK on a 1Mb block size VMFS volume grows in chunks of 1MB and so on…
• When separating OS from Data it is important to select the same block size for both VMFS volumes as other wise it might be impossible to create snapshots.
• When using a virtual RDM for Data the OS VMFS volume must have an appropriate block size. In other words the maximum file size must match the RDM size.
• When growing a VMFS volume there is no way to increase the block size and maybe you will need to grow the volume to grow the VMDK. Which might possibly be beyond the limit of the maximum file size.

My recommendation would be to forget about the block size. Make your life easier and standardize, go big and make sure you have the flexibility you need now and in the future.

1. The default of 256Mhz when no reservations are set is too conservative in my environment. What happens if you set a 100Mhz reservation?
   Nothing. The minimum value VMware HA uses to calculate with is 256Mhz. Keep in mind that this goes for slots and when using a percentage based admission control policy. Of course this can be overruled with an advanced setting (das.slotCpuInMHz) but I don’t recommend doing this.
2. What happens if you have an unbalanced cluster and the largest host fails?
   If your admission control policy is based on amount of host failures VMware HA will take this into account. However, when you select a percentage this is not the case. You will need to make sure that you specify a percentage which is equal or preferably larger than the percentage of resources provided by the largest host in this cluster. Otherwise there’s a chance that VMware HA can’t restart all virtual machines.
   
3. What would your recommendation be, reserve a specific percentage or set the amount of host failures VMware HA can tolerate?
   It depends. Yes I know, that is the obvious answer but it actually does. There are three options and each have it’s own advantages and disadvantages. Here you go:
   • Amount of host failures
     Pros: Fully automated, when a host is added to a cluster HA calculates how many slots are available.
     Cons: Can be very conservative and inflexible when reservations are used as the largest reservation dictates slot sizes.
   • Percentage reserved
     Pros: Flexible. Although reservations have its effect on the amount of available resources it impacts the environment less.
     Cons: Manual calculations need to  be done when adding additional hosts in a cluster. Unbalanced clusters can be a problem when chosen percentage is too low.
   • Designated failover host
     Pros: What you see is what you get.
     Cons: What you see is what you get.

With vSphere VMware introduced a percentage next to an amount of host failures. The percentage avoids the slot size issue as it does not use slots for admission control. So what does it use?

When you select a specific percentage that percentage of the total amount of resources will stay unused for HA purposes. First of all VMware HA will add up all available resources to see how much it has available. Then VMware HA will calculate how much resources are currently consumed by adding up all reservations of both memory and cpu for powered on virtual machines. For those virtual machines that do not have a reservation a default of 256Mhz will be used for CPU and a default of 0MB+memory overhead will be used for Memory. (Amount of overhead per config type can be found on page 28 of the resource management guide.)

In other words:

((total amount of available resources – total reserved VM resources)/total amount of available resources)
Where total reserved VM resources include the default reservation of 256Mhz and the memory overhead of the VM.

Let’s use a diagram to make it a bit more clear:

Total cluster resources are 24Ghz(CPU) and 96GB(MEM). This would lead to the following calculations:

((24Ghz-(2Gz+1Ghz+256Mhz+4Ghz))/24Ghz) = 69 % available
((96GB-(1,1GB+114MB+626MB+3,2GB)/96GB= 85 % available

As you can see the amount of memory differs from the diagram. Even if a reservation has been set the amount of memory overhead is added to the reservation. For both metrics HA admission control will constantly check if the policy has been violated or not. When one of either two thresholds are reached, memory or CPU, admission control will disallow powering on any additional virtual machines. Pretty simple huh?!

Sizing

Sizing is one of the most difficult parts in my opinion. As of vSphere the minimum requirements of vCenter have changed but it goes against my personal opinion on this subject. My recommendation would be to always start with 1 vCPU for environments with less than 10 hosts for instance. Here’s my suggestion:

• < 10 ESX Hosts
  • 1 x vCPU
  • 3GB of memory
  • Windows 64Bit OS(preferred) or Windows 32Bit OS
• > 10 ESX Hosts but < 50 ESX Hosts
  • 2 x vCPU
  • 4GB of memory
  • Windows 64Bit OS(preferred) or Windows 32Bit OS
• > 50 ESX hosts but < 200 ESX Hosts
  • 4 x vCPU
  • 4GB of memory
  • Windows 64Bit OS(preferred) or Windows 32Bit OS
• > 200 ESX Hosts
  • 4 x vCPU
  • 8GB of memory
  • Windows 64Bit OS(requirement)

My recommendation differ from VMware’s recommendation. The reason for this is that in small environments(<10 Hosts) there’s usually more flexibility for increasing resources in terms of scheduling down time. Although 2 vCPUs are a requirement I’ve seen multiple installations where a single vCPU was more than sufficient. Another argument for starting with a single vCPU would be “Practice What You Preach”. (How many times have you convinced an application owner to downscale after a P2V?!) I do however personally prefer to always use a 64Bit OS to enable upgrades to configs with more than 4GB of memory when needed.

vCenter Server in a HA/DRS Cluster

1. Disable DRS(Change Automation Level!) for your vCenter Server and make sure to document where the vCenter Server is located (My suggestion would be the first ESX host on the cluster).
2. Make sure HA is enabled for your vCenter Server, and set the startup priority to high. (Default is medium for every VM.)
3. Make sure the vCenter Server VM gets enough resources by setting the shares for both Memory and CPU to “high”.
4. Make sure other services and servers on which vCenter depends are also starting automatically, with a high priority and in the correct order like:
   1. Active Directory.
   2. DNS.
   3. SQL.
5. Write a procedure to boot the vCenter / AD / DNS / SQL manually in case of a complete power outage occurs.

Most of these recommendations are pretty obvious but you would be surprised how many environments I’ve seen where for instance MS SQL had a medium startup priority and vCenter a high priority. Or where after a complete power outage no one knows how to boot the vCenter Server. Documenting standard procedures is key here; especially know that with vSphere vCenter is more important than ever before.

Source:

http://kb.vmware.com/kb/1009080

http://kb.vmware.com/kb/1009039
ESX and vCenter Server Installation Guide
Upgrade Guide