First of all, and this is probably the most heard complaint, is that the upcoming Tintri VMstore devices will be available in a dual controller configuration which makes it more interesting to many of you probably. Especially the more up-time sensitive environments will appreciate this, and who isn’t sensitive about up-time these days? Especially in a virtualized environment where many workloads share a single device this improvement is more than welcome! The second thing which I really liked is how they enhanced their dashboard. Now this seems like a minor thing but I can ensure you that it will make your life a lot easier. Let me dump a screenshot first and then discuss what you are looking at.

The screenshot shows the per VM latency statistics… Now what is exciting about that? Well if you look at the bottom you will see the different colors and each of those represent a specific type of latency. Lets assume your VM experiences 40ms of latency and your customer starts complaining. The main thing to figure out is what causes this slow down. (Or in many cases, who can I blame?) Is your network saturated? Is the host swamped? Is it your storage device? In order to identify these types of problems you would need a monitor tool and most likely multiple tools to pinpoint the issue. Tintri decided to hook in to vCenter and just pull down the various metrics and use this to create the nice graph that you see in the screenshot. This allows you to quickly pinpoint the issue from a single pane of glass. And yes you can also expect this as a new tab within vCenter.

Another great feature which Tintri offers is the ability to realign your VMDKs. Tintri does this, unlike most solutions out there, from the “inside”. With that meaning that their solution is incorporated into their appliance and not a separate tool which needs to run against each and every VM. Smart solution which can and will safe you a lot of time.

It’s all great and amazing isn’t it? Or are there any caveats? One thing I still feel needs to be addressed is replication. With this next release it is not available yet but is that a problem now that SRM offers vSphere Replication? I guess that relieves some of the immediate pressure but I would still like to see a native Tintri’s solution providing a-sync and sync replication. Yes it will take time but I would expect though that Tintri is working on this. I tried to persuade them to make a statement yesterday they unfortunately couldn’t say anything with regards to a timeline / roadmap.

Definitely a booth I will be checking out at VMworld.

1. Live Blog: Raising The Bar, Part V
2. 5 is the magic number
3. Hot of the press: vSphere 5.0 Clustering Technical Deepdive
4. vSphere 5.0: Storage DRS introduction
5. vSphere 5.0: What has changed for VMFS?
6. vSphere 5.0: Storage vMotion and the Mirror Driver
7. Punch Zeros
8. Storage DRS interoperability
9. vSphere 5.0: UNMAP (vaai feature)
10. vSphere 5.0: ESXCLI
11. ESXi 5: Suppressing the local/remote shell warning
12. Testing VM Monitoring with vSphere 5.0
13. What’s new?
14. vSphere 5:0 vMotion Enhancements
15. vSphere 5.0: vMotion enhancement, tiny but very welcome!
16. ESXi 5.0 and Scripted Installs
17. vSphere 5.0: Storage initiatives
18. Scale Up/Out and impact of vRAM?!? (part 2)
19. HA Architecture Series – FDM (1/5)
20. HA Architecture Series – Primary nodes? (2/5)
21. HA Architecture Series – Datastore Heartbeating (3/5)
22. HA Architecture Series – Restarting VMs (4/5)
23. HA Architecture Series – Advanced Settings (5/5)
24. VMFS-5 LUN  Sizing
25. vSphere 5.0 HA: Changes in admission control
26. vSphere 5 – Metro vMotion
27. SDRS and Auto-Tiering solutions – The Injector

Once again if there it something you feel I should be covering let me know and I’ll try to dig in to it. Preferably something that none of the other blogs have published of course.

SDRS allows the aggregation of multiple datastores into a single object called a datastore cluster. SDRS will make recommendations to balance virtual machines or disks based on I/O and space utilization and during virtual machine or virtual disk provisioning make recommendations for placement. SDRS can be set in fully automated or manual mode. In manual mode SDRS will only make recommendations, in fully automated mode these recommendations will be applied by SDRS as well. When balancing recommendations are applied Storage DRS is used to move the virtual machine.

So what about Auto-Tiering solutions? Auto-tiering solutions move “blocks” around based hotspots. Yes, again, when SvMotion would migrate the virtual machine or virtual disk this process would be reset. In other words the full disk will land on the same tier and the array will need to decide at some point what belongs where… but is this an issue? In my opinion it probably isn’t but it will depend on why SDRS decides to move the virtual machine as it might lead to a temporary decrease in performance for specific chunks of data within the VM. As auto-tiering solutions help preventing performance issues by moving blocks around you might not want to have SDRS making performance recommendations but why… what is the technical reason for this?

As stated SDRS uses I/O and space utilization for balancing… Space makes sense I guess but what about I/O… what does SDRS use, how does it know where to place a virtual machine or disk? Many people seem to be under the impression that SDRS simply uses average latency but would that work in a greenfield deployment where no virtual machines are deployed yet? It wouldn’t and it would also not say much about the performance capabilities of the datastore. No in order to ensure the correct datastore is selected  SDRS needs to know what the datastore is capable off, it will need to characterize the datastore and in order to do so it uses Storage IO Control (hereafter SIOC), more specifically what we call “the injector”. The injector is part of SIOC and is a mechanism which is used to characterize each of the datastore by injecting random (read) I/O. Before you get worried, the injector only injects I/O when the datastore is idle. Even when the injector is busy and it notices other activity on the datastore it will back down and retry later. Now in order to characterize the datastore the injector uses different amount of outstanding I/Os and measures the latency for these I/Os. For example it starts with 1 outstanding I/O and gets a response within 3 miliseconds. When 3 outstanding I/Os are used the average latency for these I/Os is 3.8 miliseconds. With 5 I/Os the average latency is 4.3 and so on and so forth. For each device the outcome can be plotted as show in the below screenshot and the slope of the graph indicates the performance capabilities of the datastore. The steeper the line the lower the performance capabilities. The graphs shows the test where a multitude of datastores are characterized each being backed by a different number of spindles. As clearly shown there is a relationship between the steepness and the number of spindles used.

So why does SDRS care? Well in order to ensure the correct recommendations are made each of the datastores will be characterized in other words a datastore backed by 16 spindles will be a more logical choice than a datastore with 4 spindles. So what is the problem with Auto-Tiering solutions? Well think about it for a second… when a datastore has many hotspots an auto-tiering solution will move chunks around. Although this is great for the virtual machine it also means that when the injector characterizes the datastore it could potentially read from the SSD backed chunks or the SATA backed chunks and this will lead to unexpected results in terms of average latency and as you can imagine this will be confusing to SDRS and possibly lead to incorrect recommendations. Now, this is typically one of those scenarios which requires extensive testing and hence the reason VMware refers to the storage vendor for their recommendation around using SDRS in combination with auto-tiering solutions. My opinion: Use SDRS Space Balancing as this will help preventing downtime related to “out of space” scenarios and also help speeding up the provisioning process. On top of that you will get Datastore Maintenance Mode and Affinity Rules.

• VMFS-5
• vSphere Storage APIs – Array Integration aka VAAI
• vSphere Storage APIs – Storage Awareness aka VASA
• Profile-Driven Storage (VM Storage Profiles in the GUI)
• Storage I/O Control
• Storage DRS

I wrote separate articles about all of these features and hopefully you have read them and already see the big picture. If you don’t than this is a good opportunity to read them or head over to the vSphere Storage Blog for more details on some of these.. I guess the best way to explain it is by using an example of what life could be like when using all of these new or enhanced features compared to what is used to be like:

The Old Way: Mr Admin is managing a large environment and currently has 300 LUNs each being 500GB divided across three 8 hosts clusters. He is maintaining a massive spreadsheet with storage characteristics and runs scripts to validate virtual machines are place on the correct tier of storage. He is leveraging SIOC to avoid the noisy neighbor problem and leveraging the VAAI primitives to offload some of the tasks to the array. Still he spends a lot of time waiting, monitoring, managing virtual machines and datastores.

vSphere 5.0: Mr Admin is managing a large environment and currently has 60 thin provisioned 2.5TB LUNs presented to a single cluster. Mr Admin defines several storage tiers using VM Storage Profiles detailing storage characteristics provided through VASA. Per tier based on the information provided through VASA a Datastore Cluster is created. Datastore Clusters form the basis of Storage DRS and Storage DRS will be responsible for initial placement and preventing both IO and diskspace bottlenecks in your environment. As Storage IO Control is automatically enabled when SDRS IO balancing is enabled the noisy neighbor problem will also be eliminated. When provisioning a new virtual machine Mr Admin simple picks the appropriate VM Storage Profile and selects the compliant Datastore Cluster. If in any case Storage DRS would move things around, the “Reclaim Dead Space” feature of VAAI is used to unmap the blocks from the source datastore so that these can be re-used if and when needed.

No more spreadsheets, no extensively monitoring diskspace / latency, no more manual validation of virtual machine placement… It is all about ease of management, reducing operational effort and offloading tasks to vCenter or even your storage array!

Now you might wonder when you would need this, but think about it for a second.. what happens when you enable Storage DRS? Indeed, virtual machines might be moved around. When a virtual machine is migrated from a thin provisioned LUN to a different LUN you probably would like to reclaim the blocks that were originally allocated by the array to this volume as they are no longer needed for the source LUN. That is what unmap does. Now of course not only when a virtual machine is storage vmotioned but also when a virtual machine or for instance a virtual disk is deleted. Now one thing I need to point out that this is about unmapping blocks associated to a VMFS volume, if you delete files within a VMDK those blocks will not be unmapped!

When playing around with this I had a question from one of my colleagues, he did not have the need to unmap blocks from these thin-provisioned LUNs so he asked if you could disable it, and yes you can:

esxcli system settings advanced set --int-value 1 --option /VMFS3/EnableBlockDelete

The cool thing is that it works with net-new VMFS-5 volumes but also with upgraded VMFS-3 to VMFS-5 volumes:

1. Open the command line and go to the folder of the datastore:
   cd /vmfs/volumes/datastore_name
2. Reclaim a percentage of free capacity on the VMFS5 datastore for the thin-provisioned device by running:
   vmkfstools -y <value>


The value should be between 0 and 100, with 60 being the maximum recommended value. I ran it on a thin provisioned LUN with 60% as the percentage to reclaim. Unfortunately I didn’t have access to the back-end of the array so could not validate if any disk space was reclaimed.

/vmfs/volumes/4ddea74d-5a6eb3bc-f95e-0025b5000217 # vmkfstools -y 60
Attempting to reclaim 60% of free capacity 3.9 TB on VMFS-5 file system 'tm-pod04-sas600-sp-4t'.
Done.
/vmfs/volumes/4ddea74d-5a6eb3bc-f95e-0025b5000217 #

• VMFS-5 uses GPT instead of MBR
• VMFS-5 supports volumes up to 64TB
  • This includes Pass-through RDMs!
• VMFS-5 uses a Unified Blocksize –> 1MB
• VMFS-5 uses smaller Sub-Blocks
  • ~30.000 8KB blocks versus ~3000 64KB blocks with VMFS-3
• VMFS-5 has support for very small files (1KB)
• Non-disruptive upgrade from VMFS-3 to VMFS-5
• ATS locking enhancements (as part of VAAI)

Although some of these enhancements seem to be “minor” I beg to differ. These enhancements and new capabilities will reduce the amount of volumes needed in your environment and will increase the VM-to-Volume density ultimately leading to less management! Yes I can hear the skeptics thinking “do I really want to introduce such a large failure domain, my standard is a 500GB LUN”. Think about it for a second, although that standard might have been valid years ago, it probably isn’t today. The world has changed, recovery times have decreased, disks continue to grow, locking mechanisms have been improved and can be offloaded through VAAI. Max 10 VMs on a volume? I don’t think so!

Paul Maritz is taking the stage… Taking the next step in towards the more automated world.

We need to make infrastructure become something that people can depend on and focus on what is important to their business. Navigating your way forward offering a more dynamic infrastructure that will support your existing applications. Using a more flexible infrastructure, allowing people to take resources and aggregate to larger pools reducing operational costs by automating the use of these resources. More and more use of social media and use of mobile devices to connect anytime anywhere and most importantly securely.

Today we will be talking about  a more efficient infrastructure with exists of three stages IT Production, Business Production and IT as a Service. In 2009, the VI 3 era, 30% of the workloads were virtualized…. in 2010 with vSphere 4 we reached 40% and it is expected that in 2011 we will hit 50% virtualized with the majority on vSphere.

Accelerating and Amplifying business agility with vSphere 5! Not only vSphere 5 but the worlds first Cloud Infrastructure suite! In addition to vSphere 5 today we announce vSphere Site Recovery Manager 5 (Business Continuity), vCloud Director 1.5 (Policy, Reporting, Self-Service), vCenter Operations 1.0.1 (Monitoring and Management), vShield 5 (Security and Edge functionality).

VMware vCloud = Hybrid. Your private cloud experience needs to be similar to public cloud experience. VMware allows this through the vCloud offering and vCloud Service Providers. Trusted vCloud partners like Colt, Bluelock, Singtel, Verizon, NYSE Euronext, Softbank and CSC are some of the enablers for this.

Steve Herrod up on stage… I expect it is about to get more technical

Why do these new products matter and how do they fit together. Accelerating the adoption by increasing scalability. ESX 1.0 capable of 5000 IOps, ESX 2.0 ~ 7000 IOps, VI 3 100.000 IOps, vSphere 4 300.000 IOps and vSphere 5.0 1.000.000 IOps. Besides performance availability is key. Both HA and FT have been enhanced and of course SRM 5.0 has been released. Added to SRM 5.0 is vSphere Replication. vSphere Replication allows you to use the network to replicate between sites and different arrays. It will allow you to replicate more workloads with a lower costs. SRM is about datacenter mobility, not only for an outage but also pro-actively moving datacenters after an acquisition.

What does cloud computing really mean? Being able to order what you need and what without knowing what happens behind the scenes. IT will behind the scenes validate if they meet the consumers requirements. vCloud Director is all about Simple Self-Service. Deploy virtual machines but more importantly create new vApps and offer these in your own “app store”. The IT Cloud of the producer is all about offering agility. Virtualization enables automation in a way unheard in a physical environment.

Typically multiple tiers are offered within a cloud environment. The VMware Cloud Infrastructure enable you to do so. Intelligent Policy based Management is key with vCloud Director 1.5. Linked Clones is a very important feature to provision virtual machines “aggressively” within the system. It allows for fast provisiong and save up to 60% of storage.

Profile-Driven Storage and Storage DRS are part of vSphere 5.0. It enables you to map different arrays in to logical entities by a concept called a “datastore cluster” and link these to a profile. Virtual machines will be tagged with a profile and this allows you to validate compliancy. Storage DRS does for storage what DRS does for compute resources. Storage and Network IO Control ensures each virtual machine receives what it is entitled to.

For the SMB market a brand new shared storage appliance is introduced today: vSphere Storage Appliance 1.0. It takes vanilla servers and use local drives and present it as shared storage. It will bring agility and availability through shared storage to the SMB.

Auto-Deploy, PXE booting your ESXi hypervisor in to memory! It allows to spin-up more hosts within minutes instead of hours. Adding capacity has never been this simple?

vSphere 5 offers comprehensive security and isolation capabilities through vShield 5.0. vShield App 5 allows you to select regulations to protect sensitive data. It also enables you to get additional auditing in place.

The Cloud Infrastructure represents more than a million engineering hours, more than 100 additional capabilities, more than two million QA hours, more than 2000 partner certifications to enable this.

Rick Jackson up next discussing licensing.

Industry has traditionally licensed on physical constraints. It makes it difficult to create a cloud environment. Customers need to be able to upgrade to new hardware without having physical boundaries. No more “Cores per Proc” limits, no more “Physical RAM per host license”… vSphere introducing vRAM entitlement. Virtual RAM is the amount of virtual memory configured for a powered on virtual machine. vSphere 5 used pooled vRAM across the entire environment.

Packaging has been simplified and moving from 6 down to 5 packages. vSphere Advanced has been eliminated, all customers currently using Advanced are entitled to vSphere Enterprise.

Join us at VMworld for more details around the new product releases. 10AM virtual show, be there for more technical in-depth details!

When two or more virtual machines are issuing I/Os to the same datastore Disk.SchedNumReqOutstanding will limit the amount of I/Os that will be issued to the LUN.

So what does that mean? It took me a while before I fully got it, so lets try to explain it with an example:

You have set your queue depth for your HBA to 64 and a virtual machine is issuing I/Os to a datastore. As it is just a single VM up to 64 IOs will then end up in the device driver immediately. In most environments however LUNs are shared by many virtual machines and in most cases these virtual machines should be treated equally. When two or more virtual machines issue I/O to the same datastore DSNRO kicks in. However it will only throttle the queue depth when the VMkernel has detected that the threshold of a certain counter is reached. The name of this counter is Disk.SchedQControlVMSwitches and by default it is set to 6, meaning that the VMkernel will need to have detected 6 VM switches when handling I/O before it will throttle the queue down to the value of Disk.SchedNumReqOutstanding, by default 32. (VM Switches means that it will need to detect 6 times that the selected I/O is not coming from the same VM as the previous I/O.)

The reason the throttling happens is because the VMkernel cannot control the order of the I/Os that have been issued to the driver. Just imagine you have a VM A issuing a lot of I/Os and another, VM B, issuing just a few I/Os. VM A would end up using most of the full queue depth all the time. Every time VM B issues an I/O it will be picked quickly by the VMkernel scheduler (which is a different topic) and sent to the driver as soon as another one completes from there, but it will still get behind the 64 I/Os already in the driver, which will add significantly to it’s I/O latency. By limiting the amount of outstanding requests we will allow the VMkernel to schedule VM B’s I/O sooner in the I/O stream from VM A and thus we reduce the latency penalty for VM B.

Now that brings us to the second part of all statements out there, should we really set Disk.SchedNumReqOutstanding to the same value as your queue depth? Well in the case you want your I/Os processed as quickly as possible without any fairness you probably should. But if you have mixed workloads on a single datastore, and wouldn’t want virtual machines to incur excessive latency just because a single virtual machine issues a lot if I/Os, you probably shouldn’t.

Is that it? No not really, there are several questions that remain unanswered.

• What about sequential I/O in the case of Disk.SchedNumReqOutstanding?
• How does the VMkernel know when to stop using Disk.SchedNumReqOutstanding?

Lets tackle the sequential I/O question first. The VMkernel will allow by default to issue up to 8 sequential commands (controlled by Disk.SchedQuantum) from a VM in a row even when it would normally seem more fair to take an I/O from another VM. This is done in order not to destroy the sequential-ness of VM workloads because I/Os that happen to sectors nearby the previous I/O are handled by an order of magnitude (10x is not unusual when excluding cache effects or when caches are small compared to the disk size) faster than an I/O to sectors far away. But what is considered to be sequential? Well if the next I/O is less than 2000 sectors away from the current the I/O it is considered to be sequential (controlled by Disk.SectorMaxDiff).

Now if for whatever reason one of the VMs becomes idle you would more than likely prefer your active VM to be able to use the full queue depth again. This is what Disk.SchedQControlSeqReqs is for. By default Disk.SchedQControlSeqReqs is set to 128, meaning that when a VM has been able to issue 128 commands without any switches Disk.SchedQControlVMSwitches will be reset to 0 again and the active VM can use the full queue depth of 64 again. With our example above in mind, the idea is that if VM B is issuing very rare IOs (less than 1 in every 128 from another VM) then we still let VM B pay the high penalty on latency because presumably it is not disk bound anyway.

To conclude, now that the coin has finally dropped on Disk.SchedNumReqOutstanding I strongly feel that the advanced settings should not be changed unless specifically requested by VMware GSS. Changing these values can impact fairness within your environment and could lead to unexpected behavior from a performance perspective.

I would like to thank Thor for all the help he provided.