After my article this week about (ab) using the SRA provided through Site Recovery Manager to fail-over any LUN I expected some people reaching out to me with additional questions. One of the questions which came in more than once was “is it possible to do a test-failover of a LUN which is not managed by the SRM infra”? I guess the short answer is yes it is. The long answer is: well it depends on what your definition of a “test-failover” is. Of course booting up a physical machine from SAN while keeping the same IP etc would cause conflicts. I am also not going to show you how to re’ip your physical machines as I expect you to know this. From an SRM perspective how exciting is this?

To be honest, not really. The same concept applies. For a test-failover SRM calls the SRA by a script called “command.pl” and it feeds it XML. The following lines of XML are relevant for this exercise, but the critical one is “TestFailoverStartParameters”:

--> <TestFailoverStartParameters>
--> <ArrayId>BB005056AE32820000-server_2</ArrayId>
--> <AccessGroups>
--> <AccessGroup id="domain-c7">
--> <Initiator type="iSCSI" id="iqn.1998-01.com.vmware:localhost-11616041"/>
--> <Initiator type="iSCSI" id="iqn.1998-01.com.vmware:localhost-4a15366e"/>
--> <Initiator type="NFS" id="10.21.68.106"/>
--> <Initiator type="NFS" id="10.21.68.105"/>
--> </AccessGroup>
--> </AccessGroups>
--> <TargetDevices>
--> <TargetDevice key="fs14_T1_LUN1_BB005056AE32800000_fs10_T1_LUN1_BB005056AE32820000">
--> <AccessGroups>
--> <AccessGroup id="domain-c7"/>
--> </AccessGroups>
--> </TargetDevice>
--> </TargetDevices>
--> </TestFailoverStartParameters>
--> </Command>

Now in our case we want to fail-over a random non vSphere LUN. We will need the “initiator” (server(s)) who will need to see be able to see this LUN and we will need the LUN identifier. All of this can either be found in the SRM log files (LUN identifiers) or on the physical server (initiator details). If you would call command.pl and feed it the XML file the SRA will request the array to create a snapshot and give the host access to that snapshot. Now it is up to you to take the next steps!

It is no rocket science. Anything SRM does with the SRA you can do from the command line using command.pl and a custom XML file. As mentioned in the comments in my previous article, I know people are interested in using this for Physical Hosts… I will discuss this internally, but for now don’t come close, it is not supported!

Last week I received a question and I figured I would dive in to it this week. The question was if it is possible to fail-over LUNs using VMware Site Recovery Manager (SRM) which are not part of the Cluster which SRM “manages”. In other words, can I fail-over a LUN which is attached to a physical Windows Server or to a completely separate VMware Cluster? Before we continue, I did not hack SRM itself, neither did I make any changes to the SRA.

Lets briefly explain what SRM does normally when you go through the process of of creating a DR plan. Now this is slimmed down with only focussing on the relevant stuff for this question:

• First it will discover the devices using the Storage Replication Adapter (SRA)
• It then discovers all LUNs using the SRA
• It show the replicated LUNs containing VMs to the admin
• Admin can use these in his plan and “protect” the VMs appropriately

I decided to install SRM in a nested environment using the Celerra Uber VSA. I installed the VNX SRA and configured it and went through some of the log files just to find a piece of evidence that my plan is even possible. For Windows 2008 you can find the SRM Log Files in this location by the way:

%ALLUSERSPROFILE%\VMware\VMware vCenter Site Recovery Manager\Logs\

Other locations are documented in this KB. When I created the environment I created multiple LUNs with different sizes to make them easily recognizable. The LUN which is replicated but not exposed to our vCenter/SRM environment is 25GB and the LUN which is exposed is 30GB. This is what the log files showed me when I did a quick find on the size:

(Production) fsid=14 size=30000MB alloc=0MB dense  read-write
path=/srm01/fs14_T1_LUN1_BB005056AE32800000/fs14_T1_LUN1_BB005056AE32800000 (snapped)

(Production) fsid=16 size=25000MB alloc=0MB dense read-write
path=/vc01/fs16_T1_LUN2_BB005056AE32800000/fs16_T1_LUN2_BB005056AE32800000 (snapped)

As you can see both my 25GB and my 30GB LUN is listed. I added a name to it which also allows me to quickly identify it “srm01″ and “vc01″, where “vc01″ is the one which is not managed by SRM.

So how does SRM get this information? Well it is actually pretty straight forward, SRM calls a script which is part of the SRA. SRM feeds this script XML. This XML code contains the commands / details required. I’ve written about this a long time ago when I was troubleshooting SRM and it is still applicable:

perl command.pl < file.xml

Now the XML file is of course key here… How does that need to be structured and can we use, or should I say abuse, it to do a fail-over of a LUN which is not “managed” by SRM/vCenter. Well I started digging and it turns out to be fairly straight forward. Keep in mind the disclaimer at the top though, this is not what the SRA’s were intended for… this is purely for educational purposes and far from supported. Again the logfiles exposed a lot of details here, but I stripped it down to make it readable. This is the response from the SRA when SRM asked for details on which devices are available:

2012-01-09T12:14:53.583-08:00 [05388 verbose 'SraCommand' opID=7D6C5634-00000023] discoverDevices responded with:
--> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
--> <SourceDevice state="read-write" id="1-1">
--> <Name>fs14_T1_LUN1_BB005056AE32800000</Name>
--> <Identity>
--> <Wwn>60:06:04:8c:ab:b2:88:c0:59:40:72:24:1b:5f:77:72</Wwn>
--> </Identity>
--> <TargetDevice key="fs14_T1_LUN1_BB005056AE32800000_fs10_T1_LUN1_BB005056AE32820000"/>
--> </SourceDevice>
--> <SourceDevice state="read-write" id="1-2">
--> <Name>fs16_T1_LUN2_BB005056AE32800000</Name>
--> <Identity>
--> <Wwn>60:06:04:8c:b8:50:22:96:0c:0b:bf:d8:59:0b:a1:75</Wwn>
--> </Identity>
--> <TargetDevice key="fs16_T1_LUN2_BB005056AE32800000_fs12_T1_LUN3_BB005056AE32820000"/>
--> </SourceDevice>
--> </SourceDevices>

Now if you look at SRM and try to make a Protection Group plan you will quickly discover that only those Datastores which have a VM hosted on there can be added. This is shown in the screenshot below.

As mentioned SRM filters out the “irrelevant LUNs”, to me this LUN wasn’t irrelevant however. So what’s next? I decided to initiated a fail-over and to look at the log files. When the fail-over is initiated the following is issued by SRM, again I stripped some details to make it more readable:

--> <FailoverParameters>
--> <ArrayId>BB005056AE32820000-server_2</ArrayId>
--> <AccessGroups>
--> <AccessGroup id="domain-c7">
--> <Initiator id="iqn.1998-01.com.vmware:localhost-11616041" type="iSCSI"/>
--> <Initiator id="iqn.1998-01.com.vmware:localhost-4a15366e" type="iSCSI"/>
--> <Initiator id="10.21.68.106" type="NFS"/>
--> <Initiator id="10.21.68.105" type="NFS"/>
--> </AccessGroup>
--> </AccessGroups>
--> <TargetDevices>
--> <TargetDevice key="fs14_T1_LUN1_BB005056AE32800000_fs10_T1_LUN1_BB005056AE32820000">
--> <AccessGroups>
--> <AccessGroup id="domain-c7"/>
--> </AccessGroups>
--> </TargetDevice>
--> </TargetDevices>
--> </FailoverParameters>

I guess we should be able to work with this! Using the “discoverdevices” information and combining it with the “Failover” information I should be able to construct my own custom XML file. After creating this XML file I should be able to fail-over any LUN which is part of the selected device… What is my plan? I am planning to change the following:

• Initiator id
• TargetDevice key

I wasn’t sure if I needed to change the AccessGroup so I figured I would just test it like this. I called the script as follows:

<path to perl>\bin\perl.exe command.pl < file.xml

I watched a whole bunch of messages pass by and then looked at the Celerra when then fail-over commend was completed and noticed the following:

And of course within the “unmanaged” vCenter you can see it:

Successful fail-over of a LUN which wasn’t part of an SRM Protection Group! Yes, when you replace the Initiator ID even the masking is correctly configured. The only thing left would be either resignaturing the volume or mounting the volume. This of course depends on the OS owning the volume and the desired end result. All in all, a nice little experiment… Once again, don’t try this in your own environment, it is far from supported!

With vSphere 4.0 something changed. Although at first it might not seem substantial it actually is. As of vSphere 4.0 VMware started using ADAM. ADAM is most commonly referred to as the component which enables Linked Mode. Linked Mode gives you the opportunity to manage multiple vCenter Server from a single pane of glass.

Not only will you have a single pane of glass you will also have a central store for roles and permissions. This is key! Roles and permissions are stored in ADAM.

Lets assume you have just a single vCenter Server and are not using Linked Mode. This will not impact the way vCenter Server stores its roles and permissions… it will still use ADAM. Even when cloned daily full consistency can not be guaranteed and as such I would personally not recommend using a cold Standby vCenter Server unless you are willing to take the risks and have fully tested it.

The Design and Evaluation of a Practical System for Fault-Tolerant Virtual Machines

There are many possible ways to attempt to detect failure of the primary and backupVMs. VMware FT uses UDP heartbeating between servers that are running fault-tolerantVMs to detect when a server may have crashed. In addition, VMware FT monitors thelogging traffic that is sent from the primary to the backup VM and the acknowledgmentssent from the backup VM to the primary VM.

Hello SRM users,

The VMware vCenter Site Recovery Manager (SRM) product team is looking for product feedback on SRM deployments. If you have purchased SRM, we would like to hear from you. Your participation will be very valuable to us and the information you provide will be used to improve the SRM product going forward.

You can provide your feedback by completing the survey

The survey should take no longer than 15 minutes and will expire on June 10, 2010. Please note that this survey is for SRM customers only.

Upon completion of the survey, if you are among the 1st 1000 respondents, VMware will donate $10 per response to charity. You will also receive a link to download the electronic copy of Mike Laverick’s book “Administering VMware Site Recovery Manager 4.0″ upon completion of the survey.

We appreciate you taking the time to provide us with your valuable feedback.

Thank you,
The VMware SRM Team

Site Recovery Manager 4.0.1
File size: 104 MB

You can find the download here. This patch fixes the following issues:

• Test recovery times have been improved for ESX 4.0.1 hosts that use iSCSI arrays.
• a problem that could cause a recovery plan to hang while powering-off virtual machines at the protected site if the virtual machine’s storage goes offline while the plan is running
• Customization is now supported for virtual machines running Windows 7 and Windows 2008 R2.
• a problem that could prevent IP customization from updating the /etc/hosts file on a protected virtual machine running Linux
• a problem that could cause intermittent site disconnections when there was a firewall between the sites that was configured to close connections due to inactivity
• a problem that could cause test and recovery networks to be swapped in a recovery plan after the SRM service was restarted
• a problem that could cause datastore group calculation to fail with a “Not initialized” exception when encountering a virtual machine with an RDM device for which the lunUuid is not set
• a problem that could cause a recovered virtual machine to be deleted if an administrator manually removed it from a protection group while a recovery plan was being run
• a problem that could cause the SRM Installer to fail to update vCenter credentials when running in Repair mode
• a problem that caused the Perl installation created by SRM to be incompatible with some Perl packages. This fix eliminates the need to create the temporary Perl installation mentioned in VMware Knowledge Base article 1014232.
• a problem that could cause the SRM Service to hang when a Configure All operation configured more than 300 virtual machines
• a problem that could cause recovery plan failures with hardware iSCSI HBAs connected to Clarriion arrays

I had a disagreement with a friend about where to get SRA’s from. He was under the impression that we didn’t have the arrays in our premises for all of the SRA’s on the market and so it was OK to take an SRA from a vendor as they could test it.  The fact is we do have most, or all of  the arrays for each SRA in-house but that is actually not relevant.  It is important to only take SRA’s from the VMware web site for a different reason.  When a vendor finishes updating or writing an SRA, it is run against a special program that produces a log.  The SRA and log are sent to VMware and we check them out.  Sometimes they are sent back for improving or fixes.  This continues until the SRA passes and then it is posted on our web site.  If you took the SRA from the vendor you may accidentally get an SRA that in a week or a month we might decline and send back to be fixed.  So please, make sure you get the only safe copy of an SRA available, and that is from our web site!

1. Steps to setup VMware vCenter Site Recovery Manager 4.0 with IBM SVC
   VMware vCenter Site Recovery Manager 4.0 is a disater recovery product which uses array replication technologies to failover from one site to another. This particular document looks at one particular vendor (IBM) and one particular array model (SVC). The document will take you through the replication and snapshot setup steps.
2. Steps to setup VMware vCenter Site Recovery Manager 4.0 with EMC Celerra NAS Replications
   VMware vCenter Site Recovery Manager 4.0 is a disater recovery product which uses array replication technologies to failover from one site to another. This particular document looks at one particular vendor (EMC) and one particular array model (Celerra). This document also focuses on configuring a new feature introduced in SRM 4.0, namely NAS (NFS) replictions. The document will take you through the replication and snapshot setup steps using only the command line interface (CLI).
3. Steps to setup VMware vCenter Site Recovery Manager 4.0 with NetApp NAS Replications
   VMware vCenter Site Recovery Manager 4.0 is a disater recovery product which uses array replication technologies to failover from one site to another. This particular document looks at one particular vendor (NetApp) and uses their FAS simulator. However the steps are also applicable to their standard arrays. This document also focuses on configuring a new feature introduced in SRM 4.0, namely NAS (NFS) replictions. The document will take you through the replication and snapshot setup steps using only the command line interface (CLI).

Make sure to read them if you are implementing SRM in combination with one of the mentioned arrays!

• Site Recovery Manager 4.0 Evaluator’s Guide
  The Site Recovery Manager 4.0 Evaluator’s Guide is intended to provide Site Recovery Manager customers and evaluators a guide that walks them through the Site Recovery Manager workflow that has to be completed to allow for the successful and automated service failover from the designated Site Recovery Manager protected site to the designated Site Recovery Manager recovery site. It also provides an overview that includes the considerations and guidance to execute a failback of services from the recovery site back to the site that was originally designated as the Site Recovery Manager protected site. In addition, this guide covers the new Site Recovery Manager features – VMware vSphere support, NFS support, and shared recovery site. Evaluators can work through the exercises provided in this guide to gain a first-hand experience on operating the core and new features.
• A Quick tour of SRM 4.0
  Now that SRM 4.0 is here it seemed like a good time to take quick tour around a few of the new features and also highlight some less obvious elements of the vSphere platform that make working with SRM 4.0 simpler

HA uses the highest CPU reservation of any given VM and the highest memory reservation of any given VM. If there is no reservation a default of 256Mhz will be used for the CPU slot and the memory overhead will be used for the memory slot!

If VM1 has 2GHZ and 1024GB reserved and VM2 has 1GHZ and 2048GB reserved the slot size for memory will be 2048MB+memory overhead and the slot size for CPU will be 2GHZ.

Now how does HA calculate how many slots are available per host?

Of course we need to know what the slot size for memory and CPU is first. Then we divide the total available CPU resources of a host by the CPU slot size and the total available Memory Resources of a host by the memory slot size. This leaves us with a slot size for both memory and CPU. The most restrictive number is the amount of slots for this host. If you have 25 CPU slots but only 5 memory slots the amount of available slots for this host will be 5.

The first question I got was around unbalanced clusters. Unbalanced would for instance be a cluster with 5 hosts of which one contains substantially more memory than the others. What would happen to the total amount of slots in a cluster of the following specs:

Five hosts, each host has 16GB of memory except for one host(esx5) which has recently been added and has 32GB of memory. One of the VMs in this cluster has 4CPUs and  4GB of memory, because there are no reservations set the memory overhead of 325MB is being used to calculate the memory slot sizes. (It’s more restrictive than the CPU slot size.)

This results in 50 slots for esx01, esx02, esx03 and esx04. However, esx05 will have 100 slots available. Although this sounds great admission control rules the host out with the most slots as it takes the worst case scenario into account. In other words; end result: 200 slot cluster.

With 5 hosts of 16GB, (5 x 50) – (1 x 50), the result would have been exactly the same. To make a long story short: balance your clusters when using admission control!

The second question I received this week was around limiting the slotsizes with the advanced options das.slotCpuInMHz and/or das.slotMemInMB. If you need to use a high reservation for either CPU or Memory these options could definitely be useful, there is however something that you need to know. Check this diagram and see if you spot the problem, the das.slotMemInMB has been set to 1024MB.

Notice that the memory slotsize has been set to 1024MB. VM24 has a 4GB reservation set. Because of this VM24 spans 4 slots. As you might have noticed none of the hosts has 4 slots left. Although in total there are enough slots available; they are scattered and HA might not be able to actually boot VM24. Keep in mind that admission control does not take scattering of slots into account. It does count 4 slots for VM24, but it will not verify the amount of available slots per host.

To make sure you will always have enough slots and know what your current situation is Alan Renouf wrote an excellent script. This script reports the following:

Example Output:

Cluster        : Production
TotalSlots     : 32
UsedSlots      : 10
AvailableSlots : 22
SlotNumvCPUs   : 1
SlotCPUMHz     : 256
SlotMemoryMB   : 118

My article was a collaboration with Alan and I hope you find both article valuable. We’ve put a lot of time into making things as straight forward and simplistic as we possibly can.