Storage VMotion and moving to a Thin Provisioned disk

I was just reading this article by Vladan about Storage VMotion. He explains how you can get your unused disk space back with Storage VMotion and moving to a Thin Provisioned disk at the same time. I agree that this is one of the best new features out there. It’s easy and effective.

However, you will need to keep in mind that although it seems like disk space is not used according to the Guest OS it might have been used in the past. (An OS usually only removes the pointer to the data and not the actual data itself.) If you d0 not zero out your disk before you do the Storage VMotion and migration to a thin provisioned disk you will be copying all the “filled” blocks. This is actually the same concept as for instance a VCB full image dump, which I addressed in the beginning of 2008.

So for optimizing migrations to Thin Provisioned disks either use sdelete by Microsoft/Sysinternals or use the “shrink” option within VMware tools. Both work fine, but keep in mind they can be time consuming. You could use sdelete to script the solution and actually zero-out every disk once a week.

SRDF SRA and the SPC-2 bit

I was at a customer site helping out configuring SRM yesterday. During the configuration of the EMC SRDF SRA(Storage Replication Adapter) we ran into a weird issue. Although we could see the paired arrays with a green “okay sign” we did not see any replicated LUNs. First things I usually check in these cases are:

  1. Is the LUN already formatted as VMFS
  2. Does it hold any VMs

In this case we met both requirements. After checking all the configuration settings on the SRA side, SRM side and the SAN we noticed that the SPC-2 bit was not enabled. Of course we knew that is was a required setting according to the FC San Config Guide(page 57) but this is definitely not the kind of behavior I would expect to see when it’s disabled. Anyway I did a quick search on our internal mailing list and as it appears we were not the first to encounter these issues.

The SPC-2 bit is something that comes up every now and then, so if you’ve got EMC Symmetrix storage and you are not sure whether you have applied VMware’s recommendations please read the FC San Config Guide and avoid future problems. Please bare in mind though that when you set the SPC-2 bit you might and probably will need to re-signature the disk.

NetApp’s vSphere best practices and EMC’s SRM in a can

This week both NetApp and EMC released updated versions of documents I highly recommend to everyone interested in virtualization! Now some might think why would I want to read a NetApp document when we are an EMC shop.! Or why would I want to read an EMC document when my environment is hosted on a NetApp FAS3050. The answer is simple, although both documents contain vendor specific information there’s more to be learned from these documents because the focus is VMware products. No marketing nonsense, just the good stuff!

NetApp’s guide dives in to the basics of multipathing for instance. Especially the section on iSCSI/NFS is useful, how do I setup multiple VMkernels for load balancing and are the pros and cons. EMC’s SRM and Celerra guide includes a full how to set this up. Not only the EMC side but also the VMware SRM side of it. Like I said both documents are highly recommended!

Change the default pathing policy to round robin

I just received an email from one one of my readers, Mike Laskowski, he wanted to share the following with us:

I have over 100+ LUN’s in my environment. Round Robin is officially supported on ESX4. In the past we had a script that would manually load balance the LUN’s across FAs. ESX4 has a different way to balance the LUN’s to round robin. What you can do is build the ESX server and then in the CLI do:

esxcli nmp satp setdefaultpsp –psp VMW_PSP_RR –satp VMW_SATP_SYMM

Note: You should do this before presenting LUN’s and adding datastore. If you already have LUN’s presented and datastore added, then you do that command and then you’ll have to reboot the ESX server to take effect. This will make Round Robin the default on all LUN’s. It would take forever if you had to manually change each LUN.

THX Mike Laskowski

Please note that this example is specifically for the “SYMM” SATP. SATP stands for Storage Array Type Plugin and Symm stands for EMC DMX Symmetrix. In case you are using a different array find out what the SATP is you are using and change it accordingly.

Max amount of VMs per VMFS volume

Two weeks ago I discussed how to determine the correct LUN/VMFS size. In short it boils down to the following formula:

round((maxVMs * avgSize) + 20% )

So in other words, the max amount of virtual machines per volume multiplied by the average size of a virtual machine plus 20% for snaps and .vswp rounded up. (As pointed out in the comments if you have VMs with high amounts of memory you will need to adjust the % accordingly.) This should be your default VMFS size. Now a question that was asked in one of the comments, which I already expected, was “how do I determine what the maximum amount of VMs per volume is?”. There’s an excellent white paper on this topic. Of course there’s more than meets the eye but based on this white paper and especially the following table I decided to give it a shot:

No matter what I tried typing up, and believe me I started over a billion times, it all came down to this:

  1. Decide your optimal queue depth.
    I could do a write up, but Frank Dennenman wrote an excellent blog on this topic. Read it here and read NetApp’s Nick Triantos article as well. But in short you’ve got two options:

    • Queue Depth = (Target Queue Depth / Total number of LUNs mapped from the array) / Total number of hosts connected to a given LUN
    • Queue Depth = LUN Queue Depth / Total number of hosts connected to a given LUN

    There are two options because some vendors use a Target Queue Depth and others specifically specify a LUN Queue Depth. In the case they mention both just take the one which is most restrictive.

  2. Now that you know what your queue depth should be you, let’s figure out the rest.
    Let’s take a look at the table first. I added “mv” as it was not labeled as such in the table.
    n = LUN Queue Depth
    a = Average active SCSI Commands per server
    d = Queue Depth (from a host perspective)

    m = Max number of VMs per ESX host on a single VMFS volume
    mv = Max number VMs on shared VMFS volume

    First let’s figure out what “m”, max number of VMs per host on a single volume, should be:

    • d/a = m
      queue depth 64 / 4 active I/Os on average per VM = 16 VMs per host on a single VMFS volume

    The second one is “mv”, max number of VMs on a shared VMFS volume

    • n/a = mv
      Lun Queue Depth of 1024 / 4 active I/Os on average per VM = 256 VMs in total on a single VMFS volume but multiple hosts
  3. Now that we know “d”, “m” and “mv” it should be fairly easy to give a rough estimate of the maximum amount of VMs per LUN if you actually know what your average active I/Os number is. I know this will be your next question so my tip of today:
    Windows perfmon – average disk queue length. This contains both active and queued commands.
    For Linux this is “top” and if you are already running a virtual environment open up esxtop and take a look at “qstats”.
    Another option of course would be running Capacity Planner.

Please don’t overthink this. If you are experiencing issues there are always ways to move VMs around that’s why VMware invented Storage VMotion. Standardize your environment for ease of management and also make sure you feel comfortable about the number of “eggs in one basket”.

vSphere and vmfs-undelete

This week someone asked me during the VMTN Podcast on chat if I knew where vmfs-undelete resided in vSphere. I had a look but couldn’t find it either. A quick search gave me this:

vmfs-undelete utility is not available for ESX/ESXi 4.0
ESX/ESXi 3.5 Update 3 included a utility called vmfs-undelete, which could be used to recover deleted .vmdk files. This utility is not available with ESX/ESXi 4.0.

Workaround: None. Deleted .vmdk files cannot be recovered.

So if you are currently actively using vmfs-undelete and looking into upgrading to vSphere take this in account!

Standalone vSphere hosts and the local VMFS

On twitter @lamw just asked a question which triggered me to blog it cause I expect this is something more people will run into sooner or later.

Anyone know if you can change the default VMFS block size in ESX4 during interactive installation?

This is something that I also ran into personally a couple of weeks ago. If you install ESX 4.0 with the defaults a large VMFS volume is created that fills up the disk. This VMFS volume has a default block size of 1MB which means a file size limit of 256GB.

In the setup there’s currently no way of changing the block size. (If I’m wrong please leave a comment.) The only way to avoid this is to create two VMFS volumes. The first one will need to be created during the installation and will be the volume on which the Service Console VMDK resides. The second VMFS volume should be created after the installation and will be hosting the VMs. Although it does sound like an unnecessary step I personally think it is a good approach. This way the chance of filling up(snapshots) your VMFS partition which hosts you Service Console is very slim.