This is the session that I submitted:

Session 1359: Architecting a Cloud Infrastructure
Abstract: This session will discuss the various design considerations when architecting the foundation for every solid cloud environment: vSphere 5.0. We will start with sizing and scaling and end with some operational guidance. Different examples will be used to show the impact design considerations can have on the availability of your services.

Presenters: Duncan Epping and David Hill

Just added:

Session 1262 (Wednesday 2/12 @ 12:30pm): DR of the Cloud and to the Cloud

This session will look at DR and the cloud. Two different DR scenarios will be presented in depth – DR of the cloud and DR to the cloud. DR to the cloud is how end consumers fail over resources to a cloud provider. DR of the cloud is how you fail over cloud resources from one site to another. This session will go in depth on the consumer and provider side of the architecture. We’ll look at how to replicate the data, what applications are primary targets, how to size environments, how to maintain multi-tenancy, and what to avoid when architecting these solutions. This session is a must for anyone considering tier 1 applications for the cloud.

Presenters: Chris Colotti and Duncan Epping

I glanced over the list of sessions at PEX and I definitely highly recommend adding the following to your schedule. I tried to keep the list limited by forcing my self to only list 15, too many great sessions at PEX.

• Emad Benjamin – (1187) Virtualizing Latency Sensitive Workloads and vFabric GemFire
• Thomas Kraus – (1206) vCloud Director 1.5 solution integration
• Cormac Hogan - (1231) vSphere Storage Appliance Deep Dive
• Rob Randell - (1248) Using vShield and vCenter Configuration Manager to Achieve Better Than Physical Security for Business Critical Applications
• Mike DiPetrillo - (1260) Multi-Site Cloud Deployment How-To
• Clive Wenman - (1265) SRM 5.0 & vSphere Replication – Understanding the Use cases and Implementation Options
• Chris Colotti - (1269) Upgrading the vCloud Solution Stack in an End to End Environment
• Kamau Wanguhu - (1276) vCloud Director Networking and where VXLAN fits in
• Carter Shanklin - (1316) Using Elastic Memory for Java to pool application server memory, improve consolidation ratios, and increase app server reliability.
• Kyle Gleed - (1328) Upgrading to vSphere 5.0
• Grant Suzuki - (1349) VMware vShield App and Data Security Deep Dive
• Ken Werneburg - (1363) SRM 5 Demo – New Features in Action and Q&A
• Tom Stephens - (1374) The Last Mile in HA: Application Availability
• Justin King - (1388) Up and Running with vSphere vCenter Server Appliance (VCSA)
• Alex Fontana - (1466) Design, Deploy and Optimize Exchange 2010 on vSphere

See you in Las Vegas!

• vCloud Director virtual appliance download (You will need to register for the eval)
• VMware vCloud Director Evaluator’s Guide

If you haven’t done anything with vCloud Director before the following articles might also be worth reading, note that these are 1.0 based articles but most of the content is still valid today.

• vCloud Director Intro
• vCloud Director Demo
• vCloud Director networking part 1, 2, 3a and 3b
• vCloud Director Allocation Models

The VMware Cloud Infrastructure aims to reduce operational overhead and lower Total Cost of Ownership (TCO) by simplifying management tasks and abstracting complex processes. The focus of this architecture, as indicated by our customer requirements, is resource aggregation and isolation through the use of pools for each of the crucial pillars: network, storage and compute. Each of the three pillars will be carved in to multiple units of consumption with priority allocated based on their service level agreement. This will be achieved by leveraging core functionality offered by vSphere 5.0. Subsequently vShield App will be used to isolate each of the different type of workloads. As a hypervisor-based application-aware firewall solution, vShield App allows defining policies to logical, dynamic application boundaries (security groups) instead of physical boundaries.

This resource and security layering method will allow for a fast and safe deployment of new workloads.

Each of the different types of resources are carved up in to different groups for each of the respective workload types. A virtual machine, or vApp, will be deployed in one of the three different compute and security groups after which a specific networking group will be selected and a storage tier. Compute, Security and Network  group types are currently defined based on the different type of workloads this virtual infrastructure will host. In the future additional blocks may be added based on the requirements of the internal customers and the different types of workloads being deployed…

What Nutanix actually uses for their building block is an HPC form factor case like the one I discussed in May of this year. I wouldn’t call that revolutionary as Dell, Super Micro, HP (and others) sell these as well but market it differently (in my opinion a missed opportunity). What does make Nutanix somewhat unique is that they package it as a complete solution including a Virtual Storage Appliance they’ve created. It is not just a VSA but it appears to be a smart device which is capable of taking advantage of the SSD drives available and uses that as a shared cache distributed amongst each of the hosts and it uses multiple tiers of storage; SSD and SATA. It kind of reminds me of what Tintri does only this is a virtual appliance that is capable of leveraging multiple nodes. (I guess HP could offer something similar in a heartbeat if they bundle their VSA with the DL170e) Still I strongly believe that this is a promising concept and hope these guys are at VMworld so I can take a peak and discuss the technology behind this a bit more in-depth as I have a few questions from a design perspective…

• No 10Gbe redundancy? (according to the datasheet just a single port)
• Only 2 nics for VM traffic, vMotion, Management? (Why not just 2 10Gbe nic ports?)
• What about when the VMware cluster boundaries are reached? (Currently 32 nodes)
• Out band management ports? (could be useful to have console access)
• How about campus cluster scenarios, any constraints?
• …..

Lets see if I can get these answered over the next couple of days or at VMworld.

Paul Maritz is taking the stage… Taking the next step in towards the more automated world.

We need to make infrastructure become something that people can depend on and focus on what is important to their business. Navigating your way forward offering a more dynamic infrastructure that will support your existing applications. Using a more flexible infrastructure, allowing people to take resources and aggregate to larger pools reducing operational costs by automating the use of these resources. More and more use of social media and use of mobile devices to connect anytime anywhere and most importantly securely.

Today we will be talking about  a more efficient infrastructure with exists of three stages IT Production, Business Production and IT as a Service. In 2009, the VI 3 era, 30% of the workloads were virtualized…. in 2010 with vSphere 4 we reached 40% and it is expected that in 2011 we will hit 50% virtualized with the majority on vSphere.

Accelerating and Amplifying business agility with vSphere 5! Not only vSphere 5 but the worlds first Cloud Infrastructure suite! In addition to vSphere 5 today we announce vSphere Site Recovery Manager 5 (Business Continuity), vCloud Director 1.5 (Policy, Reporting, Self-Service), vCenter Operations 1.0.1 (Monitoring and Management), vShield 5 (Security and Edge functionality).

VMware vCloud = Hybrid. Your private cloud experience needs to be similar to public cloud experience. VMware allows this through the vCloud offering and vCloud Service Providers. Trusted vCloud partners like Colt, Bluelock, Singtel, Verizon, NYSE Euronext, Softbank and CSC are some of the enablers for this.

Steve Herrod up on stage… I expect it is about to get more technical

Why do these new products matter and how do they fit together. Accelerating the adoption by increasing scalability. ESX 1.0 capable of 5000 IOps, ESX 2.0 ~ 7000 IOps, VI 3 100.000 IOps, vSphere 4 300.000 IOps and vSphere 5.0 1.000.000 IOps. Besides performance availability is key. Both HA and FT have been enhanced and of course SRM 5.0 has been released. Added to SRM 5.0 is vSphere Replication. vSphere Replication allows you to use the network to replicate between sites and different arrays. It will allow you to replicate more workloads with a lower costs. SRM is about datacenter mobility, not only for an outage but also pro-actively moving datacenters after an acquisition.

What does cloud computing really mean? Being able to order what you need and what without knowing what happens behind the scenes. IT will behind the scenes validate if they meet the consumers requirements. vCloud Director is all about Simple Self-Service. Deploy virtual machines but more importantly create new vApps and offer these in your own “app store”. The IT Cloud of the producer is all about offering agility. Virtualization enables automation in a way unheard in a physical environment.

Typically multiple tiers are offered within a cloud environment. The VMware Cloud Infrastructure enable you to do so. Intelligent Policy based Management is key with vCloud Director 1.5. Linked Clones is a very important feature to provision virtual machines “aggressively” within the system. It allows for fast provisiong and save up to 60% of storage.

Profile-Driven Storage and Storage DRS are part of vSphere 5.0. It enables you to map different arrays in to logical entities by a concept called a “datastore cluster” and link these to a profile. Virtual machines will be tagged with a profile and this allows you to validate compliancy. Storage DRS does for storage what DRS does for compute resources. Storage and Network IO Control ensures each virtual machine receives what it is entitled to.

For the SMB market a brand new shared storage appliance is introduced today: vSphere Storage Appliance 1.0. It takes vanilla servers and use local drives and present it as shared storage. It will bring agility and availability through shared storage to the SMB.

Auto-Deploy, PXE booting your ESXi hypervisor in to memory! It allows to spin-up more hosts within minutes instead of hours. Adding capacity has never been this simple?

vSphere 5 offers comprehensive security and isolation capabilities through vShield 5.0. vShield App 5 allows you to select regulations to protect sensitive data. It also enables you to get additional auditing in place.

The Cloud Infrastructure represents more than a million engineering hours, more than 100 additional capabilities, more than two million QA hours, more than 2000 partner certifications to enable this.

Rick Jackson up next discussing licensing.

Industry has traditionally licensed on physical constraints. It makes it difficult to create a cloud environment. Customers need to be able to upgrade to new hardware without having physical boundaries. No more “Cores per Proc” limits, no more “Physical RAM per host license”… vSphere introducing vRAM entitlement. Virtual RAM is the amount of virtual memory configured for a powered on virtual machine. vSphere 5 used pooled vRAM across the entire environment.

Packaging has been simplified and moving from 6 down to 5 packages. vSphere Advanced has been eliminated, all customers currently using Advanced are entitled to vSphere Enterprise.

Join us at VMworld for more details around the new product releases. 10AM virtual show, be there for more technical in-depth details!

Storage IO Control Best Practices:

• Enable Storage IO Control on all datastores
• Avoid external access for SIOC enabled datastores
  • To avoid any interference SIOC will stop throttling, more info here.
• When multiple datastores share the same set of spindles ensure all have SIOC enabled with comparable settings and all have sioc enabled.
• Change latency threshold based on used storage media type:
  • For FC storage the recommended latency threshold is  20 – 30 MS
  • For SAS storage the recommended latency threshold is  20 – 30 MS
  • For SATA storage the recommended latency threshold is 30 – 50 MS
  • For SSD storage the recommended latency threshold is 15 – 20 MS
• Define a limit per VM for IOPS to avoid a single VM flooding the array
  • For instance limit the amount of IOPS per VM to a 1000

Here are the questions with the answers underneath in italic:

1. Q: Why is SIOC not enabled by default?
   A: As datastores can be shared between clusters, clusters could be differently licensed and as such SIOC is not enabled by default.
2. Q: If vCenter is only needed when enabling the feature, who will keep track of latencies when a datastore is shared between multiple hosts?
   A: Latency values are actually stored on the Datastore itself. From the PARDA academic paper, I figured two methods could be used for this either through network communication or as stated by using the Datastore. Notice the file “iormstat.sf” in green in the screenshot below, I guess that answers the question… the datastore itself is used to communicate the latency of a datastore. I also confirmed with Irfan that my assessment was true.
   
3. Q: Where does datastore-wide disk scheduler run from?
   A: The datastore-wide disk scheduler is essentially SIOC or also known as the “PARDA Control Algorithm” and runs on each host sharing that datastore. PARDA consists of two key components which are “latency estimation” and “window size computation”. Latency estimation is used to detect if SIOC needs to throttle queues to ensure each VM gets its fair share. Window size computation is used to calculate what this queue depth should be for your host.
4. Q: Is PARDA also responsible for throttling the VM?
   A: No, PARDA itself or better said the two major processes that form PARDA (latency estimation and window size computation) don’t control “host local” fairness, the Local scheduler (SFQ) is responsible for that.
5. Q: Can we in any way control the I/O contention in vCD VM environment (say one VM running high I/O impacting another VM running on same host/datastore)
   A: I would highly recommend to enable this in vCloud Environments to prevent storage based DoS attacks (or just noisy neighbors) and to ensure IO fairness can be preserved. This is one of the reasons VMware developed this mechanism.
6. Q: I can’t enable SIOC with an Enterprise licence – “License not available to perform the operation”. Is it Enterprise Plus only?
   A: SIOC requires Enterprise Plus
7. Q: Can I verify what the Latency is?
   A: Yes you can, go to the Host – Performance Tab and select “Datastore”, “Real Time”, select the datastore and select “Storage I/O Control normalized latency”. Please note that the unit for measurement is microseconds!
8. Q: This doesn’t appear to work in NFS?
   A: SIOC can only be enabled on VMFS volumes currently.

If you happen to be at VMworld next week, make sure to attend this session: TA8233 Prioritizing Storage Resource Allocation in ESX Based Virtual Environments Using Storage I/O Control!

Without setting the shares you can simply enable Storage I/O controls on each datastore. This will prevent any one VM from monopolizing the datatore by leveling out all requests for I/O that the datastore receives.

This is exactly the reason why I would recommend anyone who has a large environment, and even more specifically in cloud environments, to enable SIOC. Especially in very large environments where compute, storage and network resources are designed to accommodate the highest common factor it is important to ensure that all entities can claim their fair share of resource and in this case SIOC will do just that.

Now the question is how does this actually work? I already wrote a short article on it a while back but I guess it can’t hurt to reiterate thing and to expand a bit.

First a bunch of facts I wanted to make sure were documented:

• SIOC is disabled by default
• SIOC needs to be enabled on a per Datastore level
• SIOC only engages when a specific level of latency has been reached
• SIOC has a default latency threshold of 30MS
• SIOC uses an average latency across hosts
• SIOC uses disk shares to assign I/O queue slots
• SIOC does not use vCenter, except for enabling the feature

When SIOC is enabled disk shares are used to give each VM its fair share of resources in times of contention. Contention in this case is measured in latency. As stated above when latency is equal or higher than 30MS, and the statistics around this are computed every 4 seconds, the “datastore-wide disk scheduler” will determine which action to take to reduce the overall / average latency and increase fairness. I guess the best way to explain what happens is by using an example.

As stated earlier, I want to keep this post fairly simple and I am using the example of an environment where every VM will have the same amount of shared. I have also limited the amount of VMs and hosts in the diagrams. Those of you who attended VMworld session TA8233 (Ajay and Chethan) will recognize these diagrams, I recreated and slightly modified them.

The first diagram shows three virtual machines. VM001 and VM002 are hosted on ESX01 and VM003 is hosted on ESX02. Each VM has disk shares set to a value of 1000. As Storage I/O Control is disabled there is no mechanism to regulate the I/O on a datastore level. As shown in the bottom by the Storage Array Queue in this case VM003 ends up getting more resources than VM001 and VM002 while all of them from a shares perspective were entitled to the exact same amount of resources. Please note that both Device Queue Depth’s are 32, which is the key to Storage I/O Control but I will explain that after the next diagram.

As stated without SIOC there is nothing that regulates the I/O on a datastore level. The next diagram shows the same scenario but with SIOC enabled.

After SIOC has been enabled it will start monitoring the datastore. If the specified latency threshold has been reached (Default: Average I/O Latency of 30MS) for the datastore SIOC will be triggered to take action and to resolve this possible imbalance. SIOC will then limit the amount of I/Os a host can issue. It does this by throttling the host device queue which is shown in the diagram and labeled as “Device Queue Depth”. As can be seen the queue depth of ESX02 is decreased to 16. Note that SIOC will not go below a device queue depth of 4.

Before it will limit the host it will of course need to know what to limit it to. The “datastore-wide disk scheduler” will sum up the disk shares for each of the VMDKs. In the case of ESX01 that is 2000 and in the case of ESX02 it is 1000. Next the  ”datastore-wide disk scheduler” will calculate the I/O slot entitlement based on the the host level shares and it will throttle the queue. Now I can hear you think what about the VM will it be throttled at all? Well the VM is controlled by the Host Local Scheduler (also sometimes referred to as SFQ), and resources on a per VM level will be divided by the the Host Local Scheduler based on the VM level shares.

I guess to conclude all there is left to say is: Enable SIOC and benefit from its fairness mechanism…. You can’t afford a single VM flooding your array. SIOC is the foundation of your (virtual) storage architecture, use it!

ref:
PARDA whitepaper
storage i/o control whitepaper
vmworld storage drs session
vmworld storage i/o control session

If there are any questions don’t hesitate to leave a comment. Please note that I am deliberately trying to simplify things in this first article as I don’t want you to get lost in any of the layers of networking vCD offers.

Layered

Networking within vCD is built up out of 3 distinct layers.

1. External Network
2. Org Network
3. vApp Network

These three layers have been created to give the end-user the flexibility needed in a multi purpose virtual datacenter. I have depicted all three layers in the following diagram which shows the logical relationship between the layers:

Some of you technical guys might say, that’s nice but I would like to see something less abstract. I created the following diagram which depicts the different layers in a different way. The diagram shows the three layers. I created a single External Network which links to two Org Networks. These Org Networks in its turn connect to multiple VMs(Org Y) and multiple vApps(Org X).

This is just an example however that illustrates possible network connections and as can clearly be seen it can be rather complex. As demonstrated there are multiple ways to connect vApps to each other or the outside world.

Now that we know some of the basics I will break down the three layers of networking. The  first before we will discuss any of the advanced options like vShield Edge or network pools

External Network

The External Network is used for inter-Cloud connections. Or as I like to call it “your connection to the outside world”. It is the first network “object” that you create within vCD. An External Network is always backed by a Portgroup, meaning that a portgroup needs to exist within vSphere before you can create this vCD network object. This portgroup can be on a regular vSwitch, a dvSwitch or you could use Nexus 1KV. It all works, and all of them are supported!

Of course it is heavily recommended to set this portgroup up with a VLAN for layer 2 isolation, again note that this is an outbound facing connection for your Org or for multiple Orgs.

Examples of External Networks are:

• VPN to customer site
• Internet connection

As said, an external network can be shared between organizations but is typically created per organization and is your connection from or to your virtual datacenter.

I would to stress that, the external network is your exit from your virtual datacenter or your entrance!

Org Network

The second object that is created is the Org Network. The Org Network is used for intra-Cloud connections. Or as I like to call it “Cloud internal traffic”.  An Org Network is linked to an organization and can be:

• Directly connected to an External Network
• NAT/Routed connected to an External Network
• Completely Isolated

With that meaning that although an Org Network is primarily intended for internal traffic, it can be linked to an External Network to create an entry to or exit from your virtual datacenter. Note that it doesn’t necessarily need to be connected to an External Network, it could be completed isolated and used for “Cloud internal traffic” only! A use case for this would be for instance a test/dev environment where vApps will need to communicate with each other but not with the tenants back-end.

It should also be noted that the Org Network is mandatory! Every organization needs an Org Network, it is the only mandatory network object.

Just for completeness, an Org Network consumes a segment from a Network Pool when it is NAT/Routed or Isolated. A network pool is a collection of L2 networks which will be automatically consumed by vCD when needed, and what I call a segment is one of those L2 networks… basically a portgroup. I will explain Network Pools more in-depth in part 2.

When an Org Network is directly connected it is just a logical entity and physically doesn’t exist. Again in one of the following articles(part 3) I will explain what that looks like in vCenter.

vApp Network

The vApp Network kind of resembles the Org Network as it also consumes a segment from a Network Pool. The vApp Network enables you to have a vApp internal network, this could be useful for isolating specific VMs of a vApp. The vApp Network can be:

• Directly connected to an Org Network
• NAT/Routed to an Org Network
• Completely Isolated

It should be noted that the “directly connected” option for both the Org Network and the vApp Network is just a logical connection. In the back-end it will be directly connected to the layer above.

As shown in an earlier diagram and explained above a vApp can contain multiple networks. This can be used to isolate specific VMs from the outside world. An example is shown in the following diagram where only the Web Server has a connection to the Org Network and the App and Database servers are isolated but do connect to the Web server.

Summary

vCD has three different layers of networking. Each of these layers has a specific purpose. The External Network is your connection to the outside world, the Org Network is linked to a specific Organization and the vApp network only resides within a vApp.

That is it for Part 1. Part 2 will focus on the Network Pools and Part 3 will focus on what these vApp, Org and External Networks look like on a vSphere layer and some general best practices.

My tip of the day, if you want to get to know vCD really well, check vCenter every time you make a change and see what happens!

UPDATE: for a full schematic overview check Hany’s awesome diagram.