Last week I presented in Belgium at the quarterly VMUG event in Brussels. We did a Q&A and got some excellent questions. One of them was about vSphere Metro Storage Cluster (vMSC) solutions and more explicitly about Uniform vs Non-Uniform architectures. I have written extensively about this in the vSphere Metro Storage Cluster whitepaper but realized I never blogged that part. So although this is largely a repeat of what I wrote in the white paper I hope it is still useful for some of you.
<update>As of 2013 the official required bandwidth is 250Mbps per concurrent vMotion</update>
Uniform Versus Nonuniform Configurations
VMware vMSC solutions are classified in two distinct categories, based on a fundamental difference in how hosts access storage. It is important to understand the different types of stretched storage solutions because this will impact your design and operational considerations. Most storage vendors have a preference for one of these solutions, so depending on your preferred vendor it could be you have no choice. The following two main categories are as described on the VMware Hardware Compatibility List:
- Uniform host access configuration – When ESXi hosts from both sites are all connected to a storage node in the storage cluster across all sites. Paths presented to ESXi hosts are stretched across distance.
- Nonuniform host access configuration – ESXi hosts in each site are connected only to storage node(s) in the same site. Paths presented to ESXi hosts from storage nodes are limited to the local site.
We will describe the two categories in depth to fully clarify what both mean from an architecture/implementation perspective.
With the Uniform Configuration, hosts in Datacenter A and Datacenter B have access to the storage systems in both datacenters. In effect, the storage-area network is stretched between the sites, and all hosts can access all LUNs. NetApp MetroCluster is an example of this. In this configuration, read/write access to a LUN takes place on one of the two arrays, and a synchronous mirror is maintained in a hidden, read-only state on the second array. For example, if a LUN containing a datastore is read/write on the array at Datacenter A, all ESXi hosts access that datastore via the array in Datacenter A. For ESXi hosts in Datacenter A, this is local access. ESXi hosts in Datacenter B that are running virtual machines hosted on this datastore send read/write traffic across the network between datacenters. In case of an outage, or operator-controlled shift of control of the LUN to Datacenter B, all ESXi hosts continue to detect the identical LUN being presented, except that it is now accessed via the array in Datacenter B.
The notion of “site affinity”—sometimes referred to as “site bias” or “LUN locality”—for a virtual machine is dictated by the read/write copy of the datastore. For example, when a virtual machine has site affinity with Datacenter A, its read/write copy of the datastore is located in Datacenter A.
The ideal situation is one in which virtual machines access a datastore that is controlled (read/write) by the array in the same datacenter. This minimizes traffic between datacenters and avoids the performance impact of reads’ going across the interconnect. It also minimizes unnecessary downtime in case of a network outage between sites. If your virtual machine is hosted in Datacenter B but its storage is in Datacenter A you can imagine the virtual machine won’t be able to do I/O when there is a site partition.
With the Non-uniform Configuration, hosts in Datacenter A have access only to the array in Datacenter A. Nonuniform configurations typically leverage the concept of a “virtual LUN.” This enables ESXi hosts in each datacenter to read and write to the same datastore/LUN. The clustering solution maintains the cache state on each array, so an ESXi host in either datacenter detects the LUN as local. Even when two virtual machines reside on the same datastore but are located in different datacenters, they write locally without any performance impact on either of them.
Note that even in this configuration each of the LUNs/datastores has “site affinity” defined. In other words, if anything happens to the link between the sites, the storage system on the preferred site for a given datastore is the only remaining one that has read/write access to it, thereby preventing any data corruption in the case of a failure scenario. This also means that it is recommended to align virtual machine – host affinity with datastore affinity to avoid any unnecessary disruption caused by a site isolation.
I hope this helps understanding the differences between Uniform vs Non-Uniform configurations. Many more details about vSphere Metro Storage Cluster solutions, including design and operational considerations, can be found in the vSphere Metro Storage Cluster whitepaper. Make sure to read it if you are considering, or have implemented, a stretched storage solution!
wee kiong tan says
In a uniform setup, when a network partition occurs, say datacenter a is the read write storage. Vm that are hosted in datacenter b will be trigger by HA?
Same question for non uniform metro, vms that are hosted on a data store in datacenter a, will HA trigger?
Duncan says
For Uniform: yes the VM will be restarted. However the Original VM will remain running in Site B as well as HA will not kill it. (APD scenario)
For Non-Uniform: This will be recognized as a PDL scenario. The VM will be killed in Site B by ESXi and then HA will restart it in Site A
Frank says
Hi Duncan,
we are about installing a new NetApp Metro Cluster and i am reading some documents about best practice.
The document is about vsphere 5.0 u1.
Are the advanced settings like
das.maskCleanShutdownEnabled and disk.terminateVMOnPDLDefault
also valid for 5.1 or 5.1 u1?
Thanks
Frank
Duncan says
Yes 5.0 u1 and upwards
Inu Wikantiyoso says
Which one is the best for Oracle RAC ? Uniform or non-Uniform ? We need to run Oracle RAC node in both datacenter accessing the shared disk presented by MSC.
Data center software says
Very interesting. Our company has been using software for data center management that works with both uniform and non-uniform configurations. I would recommend taking a look at AssetCentral if you’re in need of a solution that works both ways.
Greg Lenczewski says
Hi Duncan,
I have read this blog post as well as your technical whitepaper on vMSC.and KB: 2032346. This is outstanding information. For those who are potentially rationalizing between vMSC and SRM like I do, I noticed you have another blog post with underlying links to other sources that are definitely worth checking out. All this information will most certainly help others decide which type of solution to implement.
Thanks,
Greg
prashant154 says
My VMs not restarting on other site when my storage gets disconnected in site. I can see iscsi paths greyed out but vm shows still powered on in that site. Even i can browse datastores in that site..After some time i get yellow traingular icon on couple of VMs in that site. I am using uniform access setup.