I figured it should be possible to achieve this using a CNAME but also recalled something around vCenter screwing this up again. I tested it anyway and with success. This is what I did:

• Added both NFS servers to DNS
• Create a CNAME (DNS Alias) and pointed to the “active” NFS server
• I used the name “nasdr” to make it obvious what it is used for
• Created an NFS share (drtest) on the NFS server
• Mount the NFS export using vCenter or though the CLI
• esxcfg-nas -a -o nasdr -s /drtest drtest
• Check the UUID using vCenter or through the CLI
• ls -lah /vmfs/volumes
• example output:
  lrwxr-xr-x    1 root     root           17 Feb  6 10:56 drtest -> e9f77a89-7b01e9fd
• Created a virtual machine on the nfsdatastore
• Enabled replication to my “standby” NFS server
• I killed my “active” NFS server environment (after validating it had completed replication)
• Changed the CNAME to point to the secondary NFS server
• Unmounted the volume old volume
• esxcfg-nas -d drtest
• I did a vmkping to “nasdr” just to validate the destination IP had changed
• Rescanned my storage using “esxcfg-rescan -A”
• Mounted the new volume
• esxcfg-nas -a -o nasdr -s /drtest drtest
• Checked the UUID using the CLI
• ls -lah /vmfs/volumes
• example output:
  lrwxr-xr-x    1 root     root           17 Feb  6 13:09 drtest -> e9f77a89-7b01e9fd
• Powered on the virtual machine now running on the secondary NFS server

As you can see, both volumes had the exact same UUID. After the fail-over I could power-on the virtual machine. No need to re-register the virtual machines within vCenter first. Before I wanted to share it with the world I reached out to my friends at NetApp. Vaughn Stewart connected me with Peter Learmonth who validated my findings and actually pointed me to a blog article he wrote about this topic. I suggest to head-over to Peter’s article for more details on this.

Currently 4 filters have been made public:

1. config.vpxd.filter.hostRescanFilter
2. config.vpxd.filter.vmfsFilter
3. config.vpxd.filter.rdmFilter
4. config.vpxd.filter.SameHostAndTransportsFilter

The first filter on the list is one I discussed roughly a year ago. The “Host Rescan Filter” makes it possible to disable the automatic storage rescan that occurs on all hosts after a VMFS volume has been created. The reason you might want to avoid this is when you adding multiple volumes and want to avoid multiple rescans but just initiate a single rescan after you create your final volume. By setting “config.vpxd.filter.hostRescanFilter” to false the automatic rescan is disabled. In short the steps needed:

1. Open up the vSphere Client
2. Go to Administration -> vCenter Server
3. Go to Settings -> Advanced Settings
4. If the key “config.vpxd.filter.hostRescanFilter” is not available add it and set it to false

To be honest this is the only storage filter I would personally recommend using. For instance “config.vpxd.filter.rdmFilter” when set to “false” will enable you to add a LUN as an RDM to a VM while this LUN is already used as an RDM by a different VM. Now that can be useful in very specific situations like when MSCS is used, but in general should be avoided as data could be corrupted when the wrong LUN is selected.

The filter “config.vpxd.filter.vmfsFilter” can be compared to the RDM filter as when set to false it would enable you to overwrite a VMFS volume with VMFS or re-use as an RDM. Again, not something I would recommend enabling as it could lead to loss of data which has a serious impact on any organization.

Same goes for “config.vpxd.filter.SameHostAndTransportsFilter”. When it is set to “False” you can actually add an “incompatible LUN” as an extend to an existing volume. An example of an incompatible LUN would for instance be a LUN which is not presented to all hosts that have access to the VMFS volume it will be added to. I can’t really think of a single reason to change the defaults on this setting to be honest besides troubleshooting, but it is good to know they are there.

Most of the storage filters have its specific use cases. In general storage filters should be avoided, except for “config.vpxd.filter.hostRescanFilter” which has proven to be useful in specific situations.

VMware published a great KB article which describes these. For instance:

NFS.HeartbeatMaxFailures
The number of consecutive heartbeat requests that must fail before the server is marked as unavailable.

The KB article does not only explain the separate NFS settings but also how you can calculate how long it can take before ESX marks a NFS share as unavailable. Good stuff, definitely highly recommended!

We checked multiple whitepapers before we went onsite and our primary source was this excellent article by Scott Drummonds. After start vscsiStats and receiving a “successful started”  we waited for 15 minutes and verified if we could see any data at all. Unfortunately we did not see anything. What is happening here? We checked the build/patch level and it was ESX 3.5 Update 4. Nothing out of the ordinary I would say. After trying several VMs we still did not see anything with “vscsiStats -s -w <worldID>”. For some weird reason, in contrary to what all blog articles are stating and what Scott Drummonds states we had to use the following command:

vscsiStats -s -t -w <worldID>

This might not be the case in most situations, but again we had to add “-t” to capture any data. You can find the world ID of the VM you want to monitor the performance by using the following command:

vscsiStats -l

After a couple of minutes you can verify if any data is being collected by using the following command:

vscsiStats -p all -w <worldID>

If you want to save your data in a CSV file to import it in Excel use the following:

vscsiStats -p all -c -w <worldID> > /tmp/vmstats-<vmname>.csv

Don’t forget to stop the monitoring:

vscsiStats -x -w <worldID>

So what’s the outcome of this all? Well with vscsiStats you can create great diagrams which for instance show the latency. This can be very useful in NFS environments as esxtop does not show this info:

If you don’t want to do this by hand, check out this article by Gabe.

This would need to go through the normal process which could take weeks. Somehow Vaughn managed to speed things up and just published the new version, 4.4, of TR-3428.

Read the document, even if you don’t own a NetApp there’s still useful info in there! In my opinion it’s one of the best VMware documents I’ve seen so far from a vendor. (Although the Cisco pdf is also a great read!)

As far as I can judge at this point in time the document seems to be also following the VMware guidelines. NetApp suggests running several commands directly on the Service Console to do the desired changes. Although I do agree with their suggested changes, I don’t agree with the order of the procedure.

On page 14 in both “procedures” NetApp suggests that you enter maintenance mode before applying the changes. Which is definitely something I would also suggest. But they also suggest to exit maintenance mode right before you reboot the server. In theory this could lead to VM’s being VMotioned to the host you’re about to reboot. When this happens the VM’s will be killed without any notice, which could lead to all sorts of problems as you can imagine.

So if you’re about to make the changes NetApp suggests, please change the order and do a reboot first and exit maintenance mode when the reboot is completed.

In the past NetApp had a best practices document which stated that it should be disabled by setting it to “1″. But, as some noticed this can and probably will result in corrupt file-systems. So this “best practice” mysteriously disappeared from the NetApp VI3 Best Practices guide and a KB Article with the VMware best practice on this setting popped up.

So if you did set “NFS.LockDisable” to 1 please change it back to “0″.  

It might be beneficial to also implement the ”prefvmx.ConsolidateDeleteNFSLocks” that Scott discussed along with patch ESX350-200808401-BG. This setting is to avoid long delays when deleting ESX snapshots. This can take up to 30 seconds, which is quite long compared to iSCSI or FC. So you should only implement this fix if you run NFS and do VMware snapshots at them same time and are experiencing these dalays.

I do recommend that everyone with an NFS filer takes a look at the NetApp best practices document because it does contain valuable information, but before you apply it besure that it doesn’t conflict with a VMware best practice!