Change the default pathing policy to round robin

10 July, 2009 By 6 Comments

I just received an email from one one of my readers, Mike Laskowski, he wanted to share the following with us:

I have over 100+ LUN’s in my environment. Round Robin is officially supported on ESX4. In the past we had a script that would manually load balance the LUN’s across FAs. ESX4 has a different way to balance the LUN’s to round robin. What you can do is build the ESX server and then in the CLI do:

esxcli nmp satp setdefaultpsp –psp VMW_PSP_RR –satp VMW_SATP_SYMM

Note: You should do this before presenting LUN’s and adding datastore. If you already have LUN’s presented and datastore added, then you do that command and then you’ll have to reboot the ESX server to take effect. This will make Round Robin the default on all LUN’s. It would take forever if you had to manually change each LUN.

THX Mike Laskowski

Please note that this example is specifically for the “SYMM” SATP. SATP stands for Storage Array Type Plugin and Symm stands for EMC DMX Symmetrix. In case you are using a different array find out what the SATP is you are using and change it accordingly.

Filed Under: Server Tagged With: , , ,

Max amount of VMs per VMFS volume

7 July, 2009 By 4 Comments

Two weeks ago I discussed how to determine the correct LUN/VMFS size. In short it boils down to the following formula:

round((maxVMs * avgSize) + 20% )

So in other words, the max amount of virtual machines per volume multiplied by the average size of a virtual machine plus 20% for snaps and .vswp rounded up. (As pointed out in the comments if you have VMs with high amounts of memory you will need to adjust the % accordingly.) This should be your default VMFS size. Now a question that was asked in one of the comments, which I already expected, was “how do I determine what the maximum amount of VMs per volume is?”. There’s an excellent white paper on this topic. Of course there’s more than meets the eye but based on this white paper and especially the following table I decided to give it a shot:

No matter what I tried typing up, and believe me I started over a billion times, it all came down to this:

  1. Decide your optimal queue depth.
    I could do a write up, but Frank Dennenman wrote an excellent blog on this topic. Read it here and read NetApp’s Nick Triantos article as well. But in short you’ve got two options:

    • Queue Depth = (Target Queue Depth / Total number of LUNs mapped from the array) / Total number of hosts connected to a given LUN
    • Queue Depth = LUN Queue Depth / Total number of hosts connected to a given LUN

    There are two options because some vendors use a Target Queue Depth and others specifically specify a LUN Queue Depth. In the case they mention both just take the one which is most restrictive.

  2. Now that you know what your queue depth should be you, let’s figure out the rest.
    Let’s take a look at the table first. I added “mv” as it was not labeled as such in the table.
    n = LUN Queue Depth
    a = Average active SCSI Commands per server
    d = Queue Depth (from a host perspective)

    m = Max number of VMs per ESX host on a single VMFS volume
    mv = Max number VMs on shared VMFS volume

    First let’s figure out what “m”, max number of VMs per host on a single volume, should be:

    • d/a = m
      queue depth 64 / 4 active I/Os on average per VM = 16 VMs per host on a single VMFS volume

    The second one is “mv”, max number of VMs on a shared VMFS volume

    • n/a = mv
      Lun Queue Depth of 1024 / 4 active I/Os on average per VM = 256 VMs in total on a single VMFS volume but multiple hosts
  3. Now that we know “d”, “m” and “mv” it should be fairly easy to give a rough estimate of the maximum amount of VMs per LUN if you actually know what your average active I/Os number is. I know this will be your next question so my tip of today:
    Windows perfmon – average disk queue length. This contains both active and queued commands.
    For Linux this is “top” and if you are already running a virtual environment open up esxtop and take a look at “qstats”.
    Another option of course would be running Capacity Planner.

Please don’t overthink this. If you are experiencing issues there are always ways to move VMs around that’s why VMware invented Storage VMotion. Standardize your environment for ease of management and also make sure you feel comfortable about the number of “eggs in one basket”.

Filed Under: Server Tagged With: , , ,

vSphere and vmfs-undelete

3 July, 2009 By 1 Comment

This week someone asked me during the VMTN Podcast on chat if I knew where vmfs-undelete resided in vSphere. I had a look but couldn’t find it either. A quick search gave me this:

vmfs-undelete utility is not available for ESX/ESXi 4.0
ESX/ESXi 3.5 Update 3 included a utility called vmfs-undelete, which could be used to recover deleted .vmdk files. This utility is not available with ESX/ESXi 4.0.

Workaround: None. Deleted .vmdk files cannot be recovered.

So if you are currently actively using vmfs-undelete and looking into upgrading to vSphere take this in account!

Filed Under: Various Tagged With: , , ,

Standalone vSphere hosts and the local VMFS

29 June, 2009 By 16 Comments

On twitter @lamw just asked a question which triggered me to blog it cause I expect this is something more people will run into sooner or later.

Anyone know if you can change the default VMFS block size in ESX4 during interactive installation?

This is something that I also ran into personally a couple of weeks ago. If you install ESX 4.0 with the defaults a large VMFS volume is created that fills up the disk. This VMFS volume has a default block size of 1MB which means a file size limit of 256GB.

In the setup there’s currently no way of changing the block size. (If I’m wrong please leave a comment.) The only way to avoid this is to create two VMFS volumes. The first one will need to be created during the installation and will be the volume on which the Service Console VMDK resides. The second VMFS volume should be created after the installation and will be hosting the VMs. Although it does sound like an unnecessary step I personally think it is a good approach. This way the chance of filling up(snapshots) your VMFS partition which hosts you Service Console is very slim.

Filed Under: Server Tagged With: ,

load balancing active/active SANs part II

26 June, 2009 By Leave a Comment

About a year ago I wrote about a script that would load balance your Active/Active SAN by evenly dividing LUNs on all available paths. A week ago I provided Kees van Vloten with this script so that it could be incorporated into a scripted install solution. Kees has enhanced the script and emailed it so that I could share it with you guys:


for N_PATHS in 2 4 6 8; do
# These are the LUNs with N_PATHS:
LUN_LIST=`esxcfg-mpath -l | egrep "^Disk.+has $N_PATHS paths" | awk '{print $2}'`
N=1
for LUN in $LUN_LIST; do
echo "LUN: $LUN, Counter: $N, Possible paths:"
esxcfg-mpath -q --lun=$LUN | grep "FC" | awk '{print $4}'
# Take the Nth path for this LUN
LUN_NEWPATH=`esxcfg-mpath -q --lun=$LUN | \
grep "FC" | awk '{print $4}' | head -n $N | tail -n 1`
# Make the Nth path the preferred path
esxcfg-mpath --lun=$LUN --path=$LUN_NEWPATH --preferred
echo ""
# Increase N (within the limit)
N=$(($N+1))
if [ $N -gt $N_PATHS ]; then
N=1
fi
done
done

Thanks for sharing,

Filed Under: Server Tagged With: , ,

VMFS/LUN size?

23 June, 2009 By 33 Comments

A question that pops up on the VMTN Community once every day is what size VMFS datastore should I create? The answer always varies,  one says “500Gb” the other says “1TB”. Now the real answer should be, it depends.

Most companies can use a simple formula in my opinion. First you should answer these questions:

  • What’s the maximum amount of VMs you’ve set for a VMFS volume?
  • What’s the average size of a VM in your environment? (First remove the really large VM’s that typically get an RDM.)

If you don’t know what the maximum amount of VMs should be just use a safe number, anywhere between 10 and 15. Here’s the formula I always use:

round((maxVMs * avgSize) + 20% )

I usually use increments of 25GB. This is where the round comes in to play. If you end up with 380GB round it up to 400GB and if you end up with 321GB round it up to 325GB. Let’s assume your average VM size is 30GB and your max amount of VMs per VMFS volume is 10:

(10*30) + 60 =360
360 rounded up -> 375GB

Filed Under: Server Tagged With: , , , ,

The file is too big…err, no it’s not

22 June, 2009 By 2 Comments

I run VMware Workstation at home because ESX doesn’t have drivers for my SATA controller. After a motherboard failure I had to reconstruct my software RAID, and this morning I tried to recreate a virtual disk I use for saving images of my laptop. Previously it was approximately 140Gb in size; after a rearrange the parition is 237Gb, so I will make the disk 236Gb. This is on an ext4 filesystem. So, off I go:

igibbs@host:/images$ vmware-vdiskmanager -c -t 0 -a ide -s 236GB fs1-images.vmdk
Creating disk '/images/fs1-images.vmdk'
Failed to create disk: The file is too big for the filesystem (0xc00000015).

Err, no it’s not. The maximum file size for ext3 and ext4 is 16TB, for ext4 will eventually be 1,048,576TB (or 1EB). To my knowledge, that’s not block size-dependent like VMFS is. Eventually it turned out that I could create a pre-allocated disk (-t 2) of 236GB but not a sparse disk (-t 0) of 236GB:

igibbs@host:/images$ vmware-vdiskmanager -c -t 2 -a ide -s 236GB fs1-images.vmdk
Creating disk 'fs1-images.vmdk'
Create: 0% done.

Hope this helps someone. I presume it’s caused by Workstation not recognising ext4 properly.

Filed Under: Desktop Tagged With: , ,
« Older Posts
Newer Posts »