Live Blog: Raising The Bar, Part V

I am live at the Launch event in San Francisco with many other bloggers, journalists and analysts. It is the 12th of July, almost 09:00 PDT and Paul Maritz is about come up on stage to talk about the Cloud Infrastructure launch. This article will be update live during the event as we go.

Paul Maritz is taking the stage… Taking the next step in towards the more automated world.

We need to make infrastructure become something that people can depend on and focus on what is important to their business. Navigating your way forward offering a more dynamic infrastructure that will support your existing applications. Using a more flexible infrastructure, allowing people to take resources and aggregate to larger pools reducing operational costs by automating the use of these resources. More and more use of social media and use of mobile devices to connect anytime anywhere and most importantly securely.

Today we will be talking about  a more efficient infrastructure with exists of three stages IT Production, Business Production and IT as a Service. In 2009, the VI 3 era, 30% of the workloads were virtualized…. in 2010 with vSphere 4 we reached 40% and it is expected that in 2011 we will hit 50% virtualized with the majority on vSphere.

Accelerating and Amplifying business agility with vSphere 5! Not only vSphere 5 but the worlds first Cloud Infrastructure suite! In addition to vSphere 5 today we announce vSphere Site Recovery Manager 5 (Business Continuity), vCloud Director 1.5 (Policy, Reporting, Self-Service), vCenter Operations 1.0.1 (Monitoring and Management), vShield 5 (Security and Edge functionality).

VMware vCloud = Hybrid. Your private cloud experience needs to be similar to public cloud experience. VMware allows this through the vCloud offering and vCloud Service Providers. Trusted vCloud partners like Colt, Bluelock, Singtel, Verizon, NYSE Euronext, Softbank and CSC are some of the enablers for this.

Steve Herrod up on stage… I expect it is about to get more technical

Why do these new products matter and how do they fit together. Accelerating the adoption by increasing scalability. ESX 1.0 capable of 5000 IOps, ESX 2.0 ~ 7000 IOps, VI 3 100.000 IOps, vSphere 4 300.000 IOps and vSphere 5.0 1.000.000 IOps. Besides performance availability is key. Both HA and FT have been enhanced and of course SRM 5.0 has been released. Added to SRM 5.0 is vSphere Replication. vSphere Replication allows you to use the network to replicate between sites and different arrays. It will allow you to replicate more workloads with a lower costs. SRM is about datacenter mobility, not only for an outage but also pro-actively moving datacenters after an acquisition.

What does cloud computing really mean? Being able to order what you need and what without knowing what happens behind the scenes. IT will behind the scenes validate if they meet the consumers requirements. vCloud Director is all about Simple Self-Service. Deploy virtual machines but more importantly create new vApps and offer these in your own “app store”. The IT Cloud of the producer is all about offering agility. Virtualization enables automation in a way unheard in a physical environment.

Typically multiple tiers are offered within a cloud environment. The VMware Cloud Infrastructure enable you to do so. Intelligent Policy based Management is key with vCloud Director 1.5. Linked Clones is a very important feature to provision virtual machines “aggressively” within the system. It allows for fast provisiong and save up to 60% of storage.

Profile-Driven Storage and Storage DRS are part of vSphere 5.0. It enables you to map different arrays in to logical entities by a concept called a “datastore cluster” and link these to a profile. Virtual machines will be tagged with a profile and this allows you to validate compliancy. Storage DRS does for storage what DRS does for compute resources. Storage and Network IO Control ensures each virtual machine receives what it is entitled to.

For the SMB market a brand new shared storage appliance is introduced today: vSphere Storage Appliance 1.0. It takes vanilla servers and use local drives and present it as shared storage. It will bring agility and availability through shared storage to the SMB.

Auto-Deploy, PXE booting your ESXi hypervisor in to memory! It allows to spin-up more hosts within minutes instead of hours. Adding capacity has never been this simple?

vSphere 5 offers comprehensive security and isolation capabilities through vShield 5.0. vShield App 5 allows you to select regulations to protect sensitive data. It also enables you to get additional auditing in place.

The Cloud Infrastructure represents more than a million engineering hours, more than 100 additional capabilities, more than two million QA hours, more than 2000 partner certifications to enable this.

Rick Jackson up next discussing licensing.

Industry has traditionally licensed on physical constraints. It makes it difficult to create a cloud environment. Customers need to be able to upgrade to new hardware without having physical boundaries. No more “Cores per Proc” limits, no more “Physical RAM per host license”… vSphere introducing vRAM entitlement. Virtual RAM is the amount of virtual memory configured for a powered on virtual machine. vSphere 5 used pooled vRAM across the entire environment.

Packaging has been simplified and moving from 6 down to 5 packages. vSphere Advanced has been eliminated, all customers currently using Advanced are entitled to vSphere Enterprise.

Join us at VMworld for more details around the new product releases. 10AM virtual show, be there for more technical in-depth details!

Disk.SchedNumReqOutstanding the story

There has been a lot of discussion in the past around Disk.SchedNumReqOutstanding and what the value should be and how it relates to the Queue Depth. Jason Boche wrote a whole article about when Disk.SchedNumReqOutstanding (DSNRO) is used and when not and I guess I would explain it as follows:

When two or more virtual machines are issuing I/Os to the same datastore Disk.SchedNumReqOutstanding will limit the amount of I/Os that will be issued to the LUN.

So what does that mean? It took me a while before I fully got it, so lets try to explain it with an example. This is basically how the VMware IO scheduler (Start-Time Fair Scheduling aka SFQ) works.

You have set your queue depth for your HBA to 64 and a virtual machine is issuing I/Os to a datastore. As it is just a single VM up to 64 IOs will then end up in the device driver immediately. In most environments however LUNs are shared by many virtual machines and in most cases these virtual machines should be treated equally. When two or more virtual machines issue I/O to the same datastore DSNRO kicks in. However it will only throttle the queue depth when the VMkernel has detected that the threshold of a certain counter is reached. The name of this counter is Disk.SchedQControlVMSwitches and by default it is set to 6, meaning that the VMkernel will need to have detected 6 VM switches when handling I/O before it will throttle the queue down to the value of Disk.SchedNumReqOutstanding, by default 32. (VM Switches means that it will need to detect 6 times that the selected I/O is not coming from the same VM as the previous I/O.)

The reason the throttling happens is because the VMkernel cannot control the order of the I/Os that have been issued to the driver. Just imagine you have a VM A issuing a lot of I/Os and another, VM B, issuing just a few I/Os. VM A would end up using most of the full queue depth all the time. Every time VM B issues an I/O it will be picked quickly by the VMkernel scheduler (which is a different topic) and sent to the driver as soon as another one completes from there, but it will still get behind the 64 I/Os already in the driver, which will add significantly to it’s I/O latency. By limiting the amount of outstanding requests we will allow the VMkernel to schedule VM B’s I/O sooner in the I/O stream from VM A and thus we reduce the latency penalty for VM B.

Now that brings us to the second part of all statements out there, should we really set Disk.SchedNumReqOutstanding to the same value as your queue depth? Well in the case you want your I/Os processed as quickly as possible without any fairness you probably should. But if you have mixed workloads on a single datastore, and wouldn’t want virtual machines to incur excessive latency just because a single virtual machine issues a lot if I/Os, you probably shouldn’t.

Is that it? No not really, there are several questions that remain unanswered.

  • What about sequential I/O in the case of Disk.SchedNumReqOutstanding?
  • How does the VMkernel know when to stop using Disk.SchedNumReqOutstanding?

Lets tackle the sequential I/O question first. The VMkernel will allow by default to issue up to 8 sequential commands (controlled by Disk.SchedQuantum) from a VM in a row even when it would normally seem more fair to take an I/O from another VM. This is done in order not to destroy the sequential-ness of VM workloads because I/Os that happen to sectors nearby the previous I/O are handled by an order of magnitude (10x is not unusual when excluding cache effects or when caches are small compared to the disk size) faster than an I/O to sectors far away. But what is considered to be sequential? Well if the next I/O is less than 2000 sectors away from the current the I/O it is considered to be sequential (controlled by Disk.SectorMaxDiff).

Now if for whatever reason one of the VMs becomes idle you would more than likely prefer your active VM to be able to use the full queue depth again. This is what Disk.SchedQControlSeqReqs is for. By default Disk.SchedQControlSeqReqs is set to 128, meaning that when a VM has been able to issue 128 commands without any switches Disk.SchedQControlVMSwitches will be reset to 0 again and the active VM can use the full queue depth of 64 again. With our example above in mind, the idea is that if VM B is issuing very rare IOs (less than 1 in every 128 from another VM) then we still let VM B pay the high penalty on latency because presumably it is not disk bound anyway.

To conclude, now that the coin has finally dropped on Disk.SchedNumReqOutstanding I strongly feel that the advanced settings should not be changed unless specifically requested by VMware GSS. Changing these values can impact fairness within your environment and could lead to unexpected behavior from a performance perspective.

I would like to thank Thor for all the help he provided.

Order of storage tiers… (via twitter @mike_laverick)

@Mike_Laverick asked a question on twitter today about something that is stated in the Cloud Computing with vCloud Director book. His question was, and no he is not dyslectic he only had 140 characters :-)

pg65. Order of storage tiers. Doesn’t that infer FC/SDD+VMFS is “race horse” and NFS “donkey”…???

Mike was referring to the following section in the book:

SLA Service Cost RTO Storage RAID Applications
Tier 0 Premium $$$$$ 20 min SSD, FC 1+0 Exchange, SQL
Tier 1 Enterprise $$$$ 1 hour FC 1+0, 5 Web servers, Sharepoint
Tier 2 Professional $$$ 2 hours iSCSI, NFS 3, 5, X Custom apps, QA
Tier 3 Basic $ 2 days NFS 3, 5, X Dev/Test

This basically states, as Mike elegantly translated, that FC/SSD is top performing storage while NFS is slow or should I say “donkey”. Mike’s comment is completely fair. I don’t agree with this table and actually did recommend changing it, somehow that got lost during the editing phase. In the first place we shouldn’t have mixed protocols with disks. Even an FC array will perform crap if you have SATA spindles backing your VMFS volumes. Secondly, there is no way you could compare these really as there are so many factors to take in to account ranging from cache to raid-level to wire speed. I guess it is still an example as clearly mentioned on page 64, nevertheless it is misleading. I would personally prefer to have listed it as follows:

SLA Service Cost RTO Protocol Disk RAID BC/DR
Tier 1 Enterprise $$$ 20 min FC 8GBps SSD 10 Sync replication
Tier 2 Professional $$ 1 hour NFS 10GBps FC 15k 6 Async Replication
Tier 3 Basic $ 1 day iSCSI 1GBps SATA 7k 5 Backup

Of course with the side note that performance is not solely dictated by the transport mechanism used, there is no reason why NFS couldn’t or shouldn’t be Tier 1 to be honest. Once again this is just an example. Thanks Mike for pointing it out,

List of VAAI capable storage arrays?

I was browsing the VMTN community and noticed a great tip from my colleague Mostafa Khalil and I believe it is worth sharing with you. The original question was: “Does anybody have a list of which arrays support VAAI (or a certain subset of the VAAI features)?”. Mostafa updated the post a couple of days back with the following reponse which also shows the capabilities of the 2.0 version of the VMware HCL:

A new version of the Web HCL will provide search criteria specific to VAAI.

As of this date, the new interface is still in “preview” stage. You can access it by clicking the “2.0 preview” button at the top of the page which is at: http://www.vmware.com/go/hcl/

  • The criteria are grouped under Features Category, Features and Plugin’s.
  • Features Category: Choice of “All” or “VAAI-Block”
  • Features: Choice of “All”, Block Zero”, “Full Copy”, “HW Assisted Locking” and more.
  • Plugin’s: Choice of “All” and any of the listed plugins.

HCL.jpg

Unfortunately there appear to be some glitches when it comes to listing all the arrays correctly, but I am confident that it will be fixed soon… Thanks Mostafa for the great tip.

Surprising results FC/NFS/iSCSI/FCoE…

I received a preliminary copy of a this report a couple of weeks ago, but since then nothing has changed. NetApp took the time to compare FC against, FCoE, iSCSI and NFS. Like most of us, probably, I still had the VI3 mindset and expected that FC would come out on top. Fact of the matter is that everything is so close, the differences are neglectable and tr-3916 shows that regardless of the type of data access protocol used you can get the same mileage. I am glad NetApp took the time to test these scenarios and used various test scenarios. It is no longer about which protocol works best, what drives the most performance… no it is about what is easiest for you to manage! Are you an NFS shop? No need to switch to FC anymore. Do you like the simplicity of iSCSI? Go for it…

Thanks NetApp for this valuable report. Although this report of course talks about NetApp it is useful material to read for all of you!