It seems that a lot of vendors are starting to update their firmware to enable virtualized workloads from the vStorage APIs for Array Integration, also known as VAAI. Not only the vendors are starting to show interest, also the bloggers are picking up on it. Hence the reason I wanted to reiterate some of the excellent details out there and wanted to make sure everyone understands what VAAI brings. Although currently there are “only” three major improvements they can and probably will make a huge difference:
- Hardware Offloaded Copy
Up to 10x faster VM deployment, cloning, Storage vMotion etc. VAAI offloads the copy task to the array, enabling the usage of native storage based mechanism resulting in a decrease of deployment time but equally important reducing the amount of data flowing between the array and server. Check this post by Bob Plankers and this one by Matt Liebowitz which clearly demonstrates the power of hardware offloaded copies! (reducing cloning from 19Minutes to 6Minutes!) - Write Same/Zero
10 x times less I/O for common tasks. Take for instance a zero-out process. It typically sends the same SCSI command several times. By enabling this option the same command will be repeated by the storage platform resulting in reduced utilization of the server while decreasing the time span of the action. - Hardware Offloaded Locking
SCSI Reservation Conflicts…. How many times have I heard that during Health Checks / Design Reviews and while troubleshooting performance related issues. Well VAAI solves those issues as well by offloading the locking mechanism to the array, also known as Atomic Test & Set aka ATS. It will more than likely reduce latency in an environment where thin-provisioned disks are used or linked clones, or even where VMware based snapshots are used. ATS removes the need to lock the full VMFS volume but instead locks a block when an update needs to occur.
One thing I wanted to point out here, which I haven’t seen mentioned yet, is that VAAI will actually allow you to have larger VMFS volumes. Now don’t get me wrong, I am not saying that you can go beyond 2TB-512b by enabling VAAI… My point is that by having VAAI enabled you will reduce the “load” on the array and on the servers. I placed quotes around load as it will not reduce the load from a VM perspective. What I am trying to get at is that many people have limited the amount of VMs per VMFS volume because of “SCSI Reservation Conflicts”. With VAAI this will change. Now you can keep your calculations “simple” and base your VMFS size on the amount of eggs you can have in a single basket and the sum of all VMs IOPS requirements.
After reading about all of this goodness I bet many of you want to use it straight away, well of course your array will need to support it first. Tomi Hakala created a nice list of all storage platforms that are currently supported and those that will be supported soon including a time frame. If your array is supported this KB explains perfectly how to enable/disable it.
I started out with saying that there are currently only three major enhancements…. that means indeed that there is more coming up in the future. Some of which I can’t discuss and others that I can as those were already mentioned at VMworld. (If you have access to TA7121 watch it!) I can’t say when they will be available or in which release, but I think it is great to know more enhancements are being worked on.
- Dead Space Reclamation
Dead space is previously written blocks that are no longer used by the VM. Currently in order to reclaim diskspace (for instance when you’ve deleted a lot of files) these blocks you will need to zero them out with for instance sdelete and then Storage vMotion the VM. Dead Space Reclamation will enable the storage system to reclaim these dead blocks by giving block liveness information. - Out-of-space conditions notifications
This is very much an improvement for day-to-day operations. It will enable notification of possible “out-of-space” conditions on both the array vendor’s tool both also within the vSphere client!
Must reads:
Chad Sakac – What does VAAI mean to you?
Bob Plankers – If you ever needed convincing about VAAI
AndreTheGiant – VAAI
VMware KB – VAAI FAQ
VMware Support Blog – VAAI changes the way storage is handled
Matt Liebowitz – Exploring the performance benefits of VAAI
Bas Raayman – What is VAAI, and how does it add spice to my life as a VMware admin?
Good post Duncan. As the VMFS limits DO increase (at which point VAAI ATS becomes even more important), larger datastores/”big thin pools” coupled with Auto-tiering/Storage DRS will make the traditional layout models really a thing of the past…
In my customer discussions recently, I do find many of them continuing to use best practice models that are very old re: number of VMs, datastore sizing, etc.
It’s going to require a concerted effort to get people en masse to realize that times have changed, and there are better ways.
BTW – I get constant questions on VMAX support for VAAI. EMC Status:
1) EMC Unified (midrange) supports VAAI today (as of FLARE 30 or later) with all 3 public APIs.
2) EMC VMAX VAAI support arrives in Enginuity 5875 which GAs in Q4 (and since there’s only a little time left, that equals “right around the corner”)
Just FYI.
We’re definitely interested in VAAI, and our SAN team is pushing up their latest array firmware upgrade so it’ll now be complete before we begin our ESXi 4.1 rollout. Like you were noting, the main area of interest for us is hardware offload locking – between that and the DRS changes to vSphere 4.1, I could see us potentially moving away from our current small clusters with set LUN sizes (500GB) to large clusters of hosts with several max size VMFS volumes, and then use DRS to sublicense specific hosts based on need.
I do have one or two questions, though: will the change from LUN-level locking via SCSI reservation to block-level locking using VAAI impact the upgrade process to 4.1? We normally upgrade 1 host in a cluster at a time, using vMotion to evacuate hosts as we upgrade. If we’re already using a VAAI-capable array and we upgrade our first host to 4.1, won’t it start locking blocks on the LUNs by default while the remaining 4.0 hosts in the cluster continue to reserve entire LUNs? Even disabling the feature and then enabling it later could be a problem, since the feature is host-specific and not tied to the cluster. I’ve just started my research but I haven’t found an answer to these questions yet.
It will only do that if you would enable those specific features within ESX(i). It is enabled so it might be a smart idea to disable it… I am also not sure what the impact is and have asked one of the developers to explain me if it could cause any issues or not in a mixed clusters and will report back as soon as I hear something. Sorry about the confusion.
I’ve just read TR-3886 from NetApp and they tested up to 128 VM’s per Datastore with VAAI enabled. They said they were limited by the 2TB limit of VMFS but this shows how powerful VAAI can be.
Great post, thanks for sharing the goodness. I am a big fan of the VAAI program. of the current APIs I think hardware-assisted locking (ATS) is probably the most beneficial for the customers as it allows SAN customers to deploy larger clusters (larger in terms of number of ESX/ESXi nodes).
I do look forward to the continual release of additional capabilities. I think we will see LUN thin provision notification and stun very soon. Heck, it appears to already be included in some form in vSphere 4.1 with NetApp. Refer to:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021272
Going forward I expect the next set of APIs to provide more granular points of integration, which in my opinion is much more interesting (from an application and/or service provider perspective) than what we have today with the initial release.
The future of VAAI and other integration programs coming to market in 2011 are going to be truly outstanding and I’m glad to be a part of this evolution of technology.
Duncan,
You mentioned in you Tuesday, November 23, 2010 at 19:06 reply that the VAAI SCSI primitives on ESX(i) are disabled by default, it’s actually incorrect. This was true during RC Beta of 4.1, but going GA, the 3 Advanced Configurations are enabled by default.
If you would like to disable them, you can use a script (here’s a blog post on how to http://www.virtuallyghetto.com/2010/07/script-automate-vaai-configurations-in.html) OR update your kickstart to automatically disable this if you’re building out ESX(i) 4.1 host.
From what I recall during a VAAI discussion in the beta, the ESX(i) 4.1 by default will try to issue the new SCSI commands and if it fails, it will revert back to the old methods. The question I never got answered, was is this done on every single SCSI operation or is this done once? I never got a true answer from the array vendors nor from VMware engineering. I do know there is a slight overhead but it’s nil per VMware Engineers, but if you don’t have VAAI enabled array, probably good idea to disable.
–William
Thanks William, I didn’t notice it. I have asked the developers what the possible impact is. If it will try VAAI based methods first for every operation or not. I will update the post as soon as I have more info,
Did you get the answer?
Hello,
in a VMWare KB article (1021976) I read that the VMFS data mover does not leverage hardware when the source and destination VMFS volumes have different block sizes.
How do you handle this in the future? Use the same Block Size for all LUNs, e.g. also 8 MB block size for smaller LUNs?
Regards
Andreas
About the block size, I’ve added some info at the top of:
http://communities.vmware.com/docs/DOC-14090
Seems that block size change speed. But I’ve done only a few tests and actually only with a storage model (PS5000XV with RAID5).
I’ve to repead with different type of storages and maybe also different type of RAID.
I always use the same blocksize for all VMFS volumes to avoid any restrictions. 8MB is the blocksize I normally use.
Duncan, Does having bigger block size have any bad impacts? My understanding is that an 8 MB block size = to an 8MB of space for every byte thats written on the vmfs. But it doesnt seem to make sense. Can you explain that please?
VMFS Blocksize is not used for Guest OS I/O, so if you write 1KB and it fits within the disk it will not expand or waste 8MB. It will only allocate an extra 8mb when the disk is thin provisioned and you need extra diskspace. So no direct extra wastage or at least neglectable.
I see. So basically it will increase 8MB in size every time it needs to write more than what it has in a thin provisioned vmdk. Do the log files get impacted by this at all? Also, does this have any effect on the thick vmdk other than how big they can get?
Old blog post, but I have a quick question. What (if any) affect does block-level de-duplication have on the effectiveness of the ATS primitive?