• Different hosts from a CPU/Memory perspective in a single cluster (different NUMA topology)
• VMs with more than 8 vCPUs

Now the thing is that the vNUMA topology is set for a given VM during the power-on. This is based on the NUMA topology of the physical host that has received the power-on request. When you move a VM to a host which has a different NUMA topology then it could result in reduced performance. This is also described in the Performance Best Practices whitepaper for vSphere 5.0. A nice example of how you can benefit from vNUMA is explained in the recently released academic paper “Performance Evaluation of HPC Benchmarks on VMware’s ESXi Server“.

I’ve never been a huge fan of mixed clusters due to complications it adds around resource management and availability, but this is definitely another argument to try to avoid it where and when possible.

$cred = Get-Credential
$sessOpt = New-WSManSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
$rsrcURI = "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2//CIM_PhysicalMemory"
foreach ($h in (Get-VMHost)) {
Write-Output $h.Name
Get-WSManInstance -ConnectionURI ("https`://" + $h.Name + "/wsman") -Authentication basic -Credential $cred -Enumerate -Port 443 -UseSSL -SessionOption $sessOpt -ResourceURI $rsrcURI | Select ElementName, @{N="Capacity (GB)";E={$_.Capacity / 1073741824.}}, MaxMemorySpeed
}

The output will look like this:

hostname01.local
ElementName    : DIMM1
Capacity (GB)  : 2
MaxMemorySpeed : 800

hostname02.local
ElementName    : DIMM1
Capacity (GB)  : 2
MaxMemorySpeed : 800

For those wondering what more you can get from CIM I would suggest reading this great article on the VMware PowerCLI blog.

• Each display type now shows the number of Worlds, VMs and vCPUs on the host on the first line. This will allow you to quickly identify why there for instance is a high %RDY.
  
• %VMWAIT is a derivitive of %WAIT, however it does not include IDLE time and only %SWPWT and “blocked”. It could for instance also be blocked when the connectivity to the storage device has failed.
  
• In the Power display there’s a new line which is PSTATE MHZ. This shows you the different clock frequencies per state. For instance “2395″ is the clock frequency of %P0 and “1596″ is the clock frequency of %P7. Please note that “%USED” is based on the base (%P0) of your CPU. %UTIL is the utilization in it’s current state (%Px), so in this case that could be 40% of %P7 (1596) which is 638.
  
• In the “Device Display” there are new stats starting with “F”, for example FCMDs, these show the failed I/Os. Fairly quick way to see if there are any I/O errors.
  
• These two new counters in the “Memory Display”, LLSWR/s / LLSWW/s, show the amount of memory being written to host cache or read from host cache. Useful when you have enabled this feature and want to know if it is actively being used. Of course there are also vCenter stats for this one.
  

I love esxtop, with 5.0 is has become even better and especially “%VMWAIT” and the PSTATE details will come in handy at some point in time!

• vMotion Architecture, Performance, and Best Practices in VMware vSphere 5

  http://www.vmware.com/resources/techresources/10202

• Understanding Memory Management in VMware vSphere 5

  http://www.vmware.com/resources/techresources/10206

• Performance Implications of Storage I/O Control–Enabled NFS Datastores in VMware vSphere 5

  http://www.vmware.com/resources/techresources/10207

• Microsoft Exchange Server 2010 Performance on vSphere 5

  http://www.vmware.com/resources/techresources/10204

• Zimbra Collaboration Server Performance on VMware vSphere 5

  http://www.vmware.com/resources/techresources/10203

• Host Power Management in VMware vSphere 5

  http://www.vmware.com/resources/techresources/10205

• VMware vCenter Update Manager 5.0 Performance and Best Practices

  http://www.vmware.com/resources/techresources/10208

Thanks NetApp for this valuable report. Although this report of course talks about NetApp it is useful material to read for all of you!

During the call the folks at Tintri offered to give a demo of how their Storage Unit works. As you know I don’t have a deep storage background like someone as Chad Sakac but I am working with storage on a day to day basis. I look at storage from a different perspective. My interest is usually around management, performance and availability.

From operational/management perspective Tintri VMstore does change things. Tintri VMstore is VM aware, which of course is easier to accomplish when you develop your own filesystem and serve it up as NFS than when using block-based storage. Tintri leverages the VMware vSphere APIs and correlates information with what they have running on top their filesystem. Now why would you want to do that? Well for instance for simple things like Storage Level snapshots per VM, try doing that on the average FC/iSCSI array and you find yourself snapshotting a full LUN or assigning dedicated LUNs to VMs. In both case not an ideal situation.

What makes VMstore special is that on top of the integration with vSphere they also do inline deduplication and compression, meaning that although a 5u VMstore node (5u includes a UPS system) offers you 8.5 TB of usable harddisk capacity it could potentially serve multitudes of that. (Depending on the type of workload of course.) But when it is doing inline dedupe and compression what about performance? Tintri VMstore offers 16 SATA drives. No not just SATA as that probably wouldn’t meet all your performance requirements, no they also offer MLC Flash aka SSD and that is where the dedupe and compression is done. In other words, in order to enable inline dedupe and compression Tintri developed a hybrid filesystem that moves data between SSD and SATA. By the way, VMstore uses RAID-6 for the 1TB of SSD drives it contains and for the 16x1TB SATA drives. If data needs to move it decides that on a pretty granular level, 4kb. Of course VMstore is smart enough to batch these transfers to optimize bandwidth.

Each VMstore node will be served up as a single NFS Share via 10GbE. Do you need more diskspace? Hook up another VMstore node and connect to the NFS Share. Other things that are simplified of course it the management of the VMstore node itself. Need to upload log files? Don’t worry, a single click sends it to the cloud over an SSL connection and  Tintri will pick it up from there. No hassling with FTP etc. Same goes for the “call back” system for support, it will upload details to the cloud and Tintri will pick it up.

When they demoed it yesterday most workloads were actually running on SSD at that moment. (The showed me their VDI environment) The cool thing is that you can actually see the performance stats on a per VM level (see screenshot below) or even per VMDK if you want to. On top of that you can also “reserve” performance for your VMs by telling VMstore that these need to be pinned to SSD.

The following is a quote from one of their customers:

Previous attempts to virtualize our Oracle Financials application had failed – as we couldn’t deliver the performance users required,” said Don St. Onge, CIO, TIBCO Software, Inc. “With Tintri VMstore, we saw a 2X performance boost which was more than enough to keep our users happy. Tintri’s unique approach to deduplication and compression lets us run the entire 1TB database instance in only 177GB of flash memory.

Now this might be slightly overstating it, like most press releases do, as I have many customers virtualizing their production tier 1 apps, but the key take away for me is the fact that they run a 1TB database in 177GB of flash and still see a performance improvement. I guess that is due to the beefy specs of the VMstore node which is literally using multiple multi core CPUs.

So in short (copy from Tintri’s press release):

• VM-aware file system designed to service I/O workloads from VMs;
• Seamless flash/disk integration with file system for smooth workload transitions and efficient use of flash capacity;
• Monitoring, control and reporting features on a per-VM and per-virtual disk basis for greater transparency in managing storage for VMs;
• Hybrid flash/disk appliances with inline deduplication and compression capabilities.

So Tintri is hot, fantastic, great… but there must be things that you feel can be improved? Well of course there are…

I would love to see even more integration with VMware. Not only make the VMstore node VM aware but also make vSphere VMstore aware. In others I would expect and love to see plugins which allow you to do most of the VM level storage management tasks within vCenter instead of through the VMstore webinterface. (Although it is a very simple interface which you can master in seconds.) Also I feel that replication is something that it needs to have. I can imagine it is part of their roadmap but I would rather see it today than tomorrow. Having the ability to enable replication per VM and than only replicate the changed and compressed “chunks” is more than welcome. It would also be great if it had Syslog capabilities so that event correlation is even easier.

My take in short: Tintri VMstore has an interesting approach on the traditional problems virtualized infrastructures are facing, by making their nodes VM aware they are looking to solve these problems. Along the way they are simplifying management and have a very competitive price. Most definitely worth investigating if their solution meets your requirements! Their website specifically calls out “Test and Development” as one of the target solutions for VMstore, I guess by now everyone know what starting with “Test and Development” brought VMware…. Keep an eye out for these guys,

vSphere VAAI Performance on the HP P4000 G2 by Barrie Seed

The results are pretty conclusive. For block zeroing on a VMDK, VAAI accelerates the operation by 4-5x

• VAAI enabled: 109 seconds
• VAAI disabled: 482 seconds
  

VAAI Awesomeness by Anders Hansen

I guess a picture says more than a thousand words. Difference in percentage for Cloning:

Difference in time for Eager Zero Thick Creation:

 

Exploring the performance benefits of VAAI by Matt Liebowitz

To the results:

Time to create a 50GB eagerzeroedthick VMDK without VAAI: 10 minutes generating approximately 750 write IOPS on the array

Time to create a 50GB eagerzeroedthick VMDK with VAAI: 1 minute 30 seconds, could not measure IOPS (more on that later)

Clearly there is a significant difference in creating the blank eagerzeroedthick VMDK.  How about when Windows 2008 R2 is installed on that VMDK and then converted to a template?  How fast can we deploy that template?

Deploying 50GB eagerzeroedthick template without VAAI: 19 minutes generating between 1,200-1,600 IOPS (half read/write, which makes sense since it has to read from and write to the same array)

Deploying 50GB eagerzeroedthick template with VAAI: 6 minutes (again, couldn’t measure IOPS)

NetApp VMware VAAI Performance Tests by Jacint Juhasz

It’s not a surprise, the trend is the same.

Operation	Enabled VAAI	Disabled VAAI
50GB VMDK creation with cluster support (zeroed)	5:09	9:36
Clone VM within datastore (LUN)	8:36	13:38
Clone VM between datastores (LUN)	8:34	14:36
Storage VMotion	9:38	14:45

With VAAI enabled, there’s no write or read rate (as there’s no read or write from the host side), but the charts shows latency around 8-10ms. With disabled VAAI the chart looks a bit different. For the VMDK creation the write rate is around 100000KBps with 160ms latency (write only, no reads). The read/write operation shows 70000KBps IO rate with 10-15ms latency.

3PAR vSphere VAAI “Write Same” Test Results: 20x performance boost by Derek Seaman

“Write Same” Without VAAI:
70GB VMDK 2 minutes 20 seconds (500MB/sec)
240GB VMDK 8 minutes 1 second (498MB/sec)
1TB VMDK 33 minutes 10 seconds (502MB/sec)

Without VAAI the ESXi 4.1 host is sending a total 500MB/sec of data through the SAN and into the 4 ports on the 3PAR. Because the T400 is an active/active concurrent controller design, both controllers can own the same LUN and distribute the I/O load. In the 3PAR IMC (InForm Management console) I monitored the host ports and all four were equally loaded around 125MB/sec.

This shows that round-robin was functioning, and highlights the very well balanced design of the T400. But this configuration is what everyone has been using the last 10 years..nothing exciting here except if you want to weight down your SAN and disk array with processing zeros. Boorrrringgg!!

Now what is interesting, and very few arrays support, is a ‘zero detect’ feature where the array is smart enough on thin provisioned LUNs to not write data if the entire block is all zeros. So in the 3PAR IMC I was monitoring the back-end disk facing ports and sure enough, virtually zero I/O. This means the controllers were accepting 500MB/sec of incoming zeros, and writing practically nothing to disk. Pretty cool!

“Write Same” With VAAI: 20x Improvement
70GB VMDK 7 seconds (10GB/sec)
240GB VMDK 24 seconds (10GB/sec)
1TB VMDK 1 minute 23 seconds (12GB/sec)

I guess it is needless to say why VAAI rocks and why when you are looking to buy new storage it is important to inform if the array is VAAI capable, and if not make sure you ask when it will support VAAI!?! VAAI isn’t just for specific workloads, VAAI was designed to reduce stress on different layers and to decrease the cost of specific actions and more importantly for you to decrease the costs of operations!

When you format a VMFS volume you can select the blocksize (1MB, 2MB, 4MB or 8MB). This blocksize is used when the hypervisor allocates storage for the  VMDKs. So when you create a VMDK on an 8MB formatted VMFS volume it will create that VMDK out of 8MB blocks and yes indeed in the case of a 1MB formatted VMFS volume it will use 1MB. Now this blocksize also happens to be the size of the extend that is used for Think Disks. In other words, every time your thin disks needs to expand it will grow in extends of 1MB. (Related to that, with a lazy-thick disk the zero-out also uses the blocksize. So when something needs to be written to an untouched part of the VMDK it will zero out using the blocksize of the VMFS volume.)

So using a thin disk in combination with a small blocksize cause more fragmentation? Yes, more than possibly it would. However the real question is if it will hurt your performance. The answer to that is: No it won’t. The reason for it being that the VMFS blocksize is totally irrelevant when it comes to Guest OS I/O. So lets assume you have an regular Windows VM and this VM is issuing 8KB writes and reads to a 1MB blocksize formatted volume, the hypervisor won’t fetch 1MB as that could cause a substantial overhead… no it would request from the array what was requested by the OS and the array will serve up whatever it is configured to do so. I guess what people are worried about the most is sequential I/O, but think about that for a second or two. How sequential is your I/O when you are looking at it from the Array’s perspective? You have multiple hosts running dozens of VMs accessing who knows how many volumes and subsequently who knows how many spindles. That sequential I/O isn’t as sequential anymore all of a sudden it is?!

<edit> As pointed out many arrays recognize sequential i/o and prefetch which is correct, this doesn’t mean that when contiguous blocks are used it is faster as fragmented blocks also means more spindles etc </edit>

I guess the main take away here is, stop worrying about VMFS it is rock solid and it will get the job done.

From top to bottom:

• Guest device driver queue depth (LSI=32, PVSCSI=64)
• vHBA (Hard coded limit: LSI=128, PVSCSI=255)
• disk.schedNumOutstanding=32 (VMKernel),
• VMkernel Device Driver (FC=32, iSCSI=128, NFS=256,  local disk=32)
• Multiple SAN/Array Queues (Check Chad’s article for more details but it includes port buffers, port queues, disk queues etc (might be different for other storage vendors))

The following is probably worth repeating or clarifying:

The PVSCSI default queue depth is 64. You can increase it to 255 if required, please note that it is a per-device queue depth and keep in mind that this would only be truly useful when it is increased all the way down the stack and the Array Controller supports it. There is no point in increasing the queuedepth on a single layer when the other layers are not able to handle it as it would only push down the delay one layer. As explained in an article a year or three ago, disk.schednumreqoutstanding is enforced when multiple VMs issue I/Os on the same physical LUN, when it is a single VM it doesn’t apply and it will be the Device Driver queue that limits it.

I hope this provides a bit more insight to how the traffic flows. And by the way, if you are worried a single VM floods one of those queues there is an answer for that, it is called Storage IO Control!

This person was referred to an article by Frank Brix Pedersen who states the following:

Ever since ESX3.0 we have used the VMFS3 filesystem and we are still using it on vSphere. What most people don’t know is that there actually is sub versions of the VMFS.

• ESX 3.0 VMFS 3.21
• ESX 3.5 VMFS 3.31 key new feature: optimistic locking
• ESX 4.0 VMFS 3.33 key new feature: optimistic IO

The good thing about it is that you can use all features on all versions. In ESX4 thin provisioning was introduced but it does need the VMFS to be 3.33. It will still work on 3.21. The changes in the VMFS is primarily regarding the handling of SCSI reservations. SCSI reservations happens a lot of times. Creation of new vm, growing a snapshot delta file or growing thin provisioned disk etc.

I want to make sure everyone realizes that this is actually not true. All the enhancements made in 3.5, 4.0 and even 4.1 are not implemented on a filesystem level but rather on a VMFS Driver level or through the addition of specific filters or even a new datamover.

Just to give an extreme example: You can leverage VAAI capabilities on a VMFS volume with VMFS filesystem version 3.21, however in order to invoke VAAI you will need the VMFS 3.46 driver. In other words, a migration to a new datastore is not required to leverage new features!