• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Yellow Bricks

by Duncan Epping

  • Home
  • Unexplored Territory Podcast
  • HA Deepdive
  • ESXTOP
  • Stickers/Shirts
  • Privacy Policy
  • About
  • Show Search
Hide Search

cloud

Storage IO Control Best Practices

Duncan Epping · Oct 19, 2010 ·

After attending Irfan Ahmad’s session on Storage IO Control at VMworld I had the pleasure to sit down with Irfan and discuss SIOC. Irfan was so kind to review my SIOC articles(1, 2) and we discussed a couple of other things as well. The discussion and the Storage IO Control session contained some real gems and before my brain resets itself I wanted to have these documented.

Storage IO Control Best Practices:

  • Enable Storage IO Control on all datastores
  • Avoid external access for SIOC enabled datastores
    • To avoid any interference SIOC will stop throttling, more info here.
  • When multiple datastores share the same set of spindles ensure all have SIOC enabled with comparable settings and all have sioc enabled.
  • Change latency threshold based on used storage media type:
    • For FC storage the recommended latency threshold is  20 – 30 MS
    • For SAS storage the recommended latency threshold is  20 – 30 MS
    • For SATA storage the recommended latency threshold is 30 – 50 MS
    • For SSD storage the recommended latency threshold is 15 – 20 MS
  • Define a limit per VM for IOPS to avoid a single VM flooding the array
    • For instance limit the amount of IOPS per VM to a 1000

VMworld Labs Europe – Open on Monday!!

Duncan Epping · Oct 10, 2010 ·

Yes, we will be open on Monday from 13:00 till 18:00! Not only that, but the for the remaining days the labs will open up at 08:00. Some might wonder why the schedule changed, the reason for it is simple the amount of VMworld registrations was so overwhelming VMware wanted to offer everyone the opportunity to get the most out of VMworld. So if you are coming down to the Bella Center on Monday to register stick around and do some labs! There are enough topics to spent your whole Monday afternoon in the Bella Center. My personal favorites definitely are:

  • VMware View 4.5 Install and Configure
  • VMware vShield
  • VMware vCloud Director Install and Configure
  • VMware vCloud Director Networking
  • VMware vSphere Performance & Tuning

But besides these 5 there are about 25 other labs, so enough to get your hands dirty.

vCD – Networking part 3 – Use case 2

Duncan Epping · Oct 6, 2010 ·

Part 1 explained the basic concepts of networking within vCD(VMware vCloud Director), Part 2 focussed on Network Pools and Part 3 focussed on a use case which was a vApp directly connected to an External routed Org Network. It took me a while to develop part 3 and I wasn’t sure if I could find the time to do another use case or not. I received a dozen requests for another use case so I decided to free up some time to help you guys out. Please read the other parts of this series before you start reading this part. Okay, let’s dive into those trenches.

vApp fenced to an Internal Org Network

Use case:

  1. Environments where vApps are copied and redeployed for  test and development purposes. There is no connection back to the customers datacenter to avoid any interference that could be cause by these test environments.

We will start with the basics. The flow of the network in this case will be:

vmware vCD cloud director networking logical diagram

Although only two different type of networks are used, this could of course result in multiple layers if and when for instance multiple vApp Networks are created. For the purpose of this exercise we will create a vApp with 3 VMs including two different networks. You could say you classical three-tier application.

  1. WEB = Web Frontend
  2. APP =Application Server
  3. DB = Databaser Server

As you can imagine we don’t need users accessing the Application or Database Server so these two will be on a separate network segment. The Web Frontend will need to be accessible though and it will need to be able to access both the Application and the Database Server. Logically speaking that will look as follows:

vmware vCD cloud director networking logical 3-tier app diagram

Please note that the Org Network doesn’t connect back to anything! This means that in order for you to connect to your WEB vm you will need to go through the vCD Portal! Of course you could still test if your web services are working by simply deploying a desktop VM with windows XP in the same Org. Now I can hear some of you think why not just a NAT-Routed Org Network, well that is something that would work as well, but than it would be really similar to what use case 1 provided and this is purely for educational purposes.

Creating the Networks

The first step of course is to ensure you have a Network Pool. If you haven’t already created, you can use Part 2 of this series as a reference.  I am assuming here you already have a network pool and will go straight to the Org Network, which is option 7 on the home screen.

vmware vCD cloud director networking screenshot

Now you will need to select the Org that this Network will belong to and then you can decide what type of network you will create. You can do this in either “Typical” or “Advanced” mode. Both will give you the same options but it is named slightly different and Advanced will only allow you to create 1 network at a time where with Typical you can create multiple. As we have used Typical in the previous use case we will use Advanced this time. We are going to create a fully isolated Org Network so we will select “Internal Organization Network”.

vmware vCD cloud director networking screenshot

Next up we will need to select a network pool. Now you might ask yourself why we will need one when the Org Network is completely isolated? Well we will need cross-host communication when vApp/VMs need to communicate with each other and don’t reside on the same host. Although it sounds very logical, it is often overseen that this is what a network pool does. It enables cross-host communication. In this case we will select the vCloud Network Isolation Network Pool.

vmware vCD cloud director networking screenshot

Now we will need to specify the IP details for this Org Network. These IP addresses will be consumed by the VMs that are configured to use the “static pool”, in our case that will be the vShield Edge device that is deployed as part of this Isolated Network (deployed for DHCP services etc) and the WEB virtual machine.

vmware vCD cloud director networking screenshot

Of course we will need to give it a name. I tend to use the name of the Org and the type of Org Network I created.

vmware vCD cloud director networking screenshot

Now we will need to build a vApp. As stated this vApp will contain multiple VMs.

vmware vCD cloud director networking screenshot

We will give it a name.

vmware vCD cloud director networking screenshot

And we will start adding multiple VMs to it. The WEB virtual machine will have 2 NICs as it will need to connect to a device outside of vApp and to two VMs inside of the vApp.

vmware vCD cloud director networking screenshot

The following two VMs “APP” and “DB” will be configured with a single NIC as they will only need to connect to each other, all contained within the vApp.

vmware vCD cloud director networking screenshot

vmware vCD cloud director networking screenshot

Now this is the part where we will assign specific network segments to the NICs. For WEB we will connect “NIC 0” to the Internal Org Network and NIC 1 will need to be connected to a vApp Network.

vmware vCD cloud director networking screenshot

This vApp Network however will need to be created first. Please note that this is a vApp network, so only available to those VMs which are part of this vApp! Again we will need to specify IP details for the VMs to consume.

vmware vCD cloud director networking screenshot

When we have done this and have given the vApp network a name we can connect the remaining VMs to the same network.

vmware vCD cloud director networking screenshot

Now in order to have multiple copies of the same vApp running within the Org we will select “Fenced” mode for the vApp which basically will deploy a vShield Edge device.

vmware vCD cloud director networking screenshot

I guess this diagram that vCD creates makes it a bit more clear what your vApp connectivity will look like:

vmware vCD cloud director networking screenshot

And if that isn’t enough you can also check the Maps functionality that vCenter offers. This will give you a great view of how this vApp is connected within vSphere.

vmware vCD cloud director networking screenshot

So what about that desktop? And what about if we have two copies of that vApp running? Well this is what the map would look like if when we have created these. On the middle left you will see the desktop that is used for testing the WEB VMs. Both WEB virtual machines can be accessed through the VSE device, which of course means that you will need to setup NAT, but we will leave an in-depth explanation around that for the next article.

vmware vCD cloud director networking screenshot

Summary

vCloud Director Network is really powerful, but as shown by this use case it can get very complex rather fast especially when you are using multiple layers. In this example we kept it simple by using an isolated network, an External NAT/Routed Org Network would have added another layer. Features like vCenter Maps will however make it easier to understand what has been created on the vSphere layer to enable these layers of networking, make sure you take advantage of functionality like this when exploring vCD!

vCD restart issues

Duncan Epping · Sep 30, 2010 ·

One of my readers had an issue with a vCD cell not starting correctly. He wanted to know how to monitor the progress of the startup of the daemon. You can do that very easy with the following command:

tail -f /opt/vmware/cloud-director/logs/cell.log

Now you should see status updates like the following pass by:

Application Initialization: 9% complete. Subsystem 'com.vmware.vcloud.common.core' started

At some point it should of course say “Application Initialized”. If it doesn’t the cell.log file should give a hint why it hasn’t been initialized. If for whatever reason the issue persists you could always check the “vcloud-vmware-watchdog.log” for details on the dump. There is a great KB article on this topic though, make sure you read it!

Now, what I wanted to use this article for is to document percentage where a start can get stuck and the possible issue that causes it. I have just a one to start with but hopefully this will grow overtime:

  • 9% – Possibly caused by DNS issues

Storage I/O Fairness

Duncan Epping · Sep 29, 2010 ·

I was preparing a post on Storage I/O Control (SIOC) when I noticed this article by Alex Bakman. Alex managed to capture the essence of SIOC in just two sentences.

Without setting the shares you can simply enable Storage I/O controls on each datastore. This will prevent any one VM from monopolizing the datatore by leveling out all requests for I/O that the datastore receives.

This is exactly the reason why I would recommend anyone who has a large environment, and even more specifically in cloud environments, to enable SIOC. Especially in very large environments where compute, storage and network resources are designed to accommodate the highest common factor it is important to ensure that all entities can claim their fair share of resource and in this case SIOC will do just that.

Now the question is how does this actually work? I already wrote a short article on it a while back but I guess it can’t hurt to reiterate thing and to expand a bit.

First a bunch of facts I wanted to make sure were documented:

  • SIOC is disabled by default
  • SIOC needs to be enabled on a per Datastore level
  • SIOC only engages when a specific level of latency has been reached
  • SIOC has a default latency threshold of 30MS
  • SIOC uses an average latency across hosts
  • SIOC uses disk shares to assign I/O queue slots
  • SIOC does not use vCenter, except for enabling the feature

When SIOC is enabled disk shares are used to give each VM its fair share of resources in times of contention. Contention in this case is measured in latency. As stated above when latency is equal or higher than 30MS, and the statistics around this are computed every 4 seconds, the “datastore-wide disk scheduler” will determine which action to take to reduce the overall / average latency and increase fairness. I guess the best way to explain what happens is by using an example.

As stated earlier, I want to keep this post fairly simple and I am using the example of an environment where every VM will have the same amount of shared. I have also limited the amount of VMs and hosts in the diagrams. Those of you who attended VMworld session TA8233 (Ajay and Chethan) will recognize these diagrams, I recreated and slightly modified them.

The first diagram shows three virtual machines. VM001 and VM002 are hosted on ESX01 and VM003 is hosted on ESX02. Each VM has disk shares set to a value of 1000. As Storage I/O Control is disabled there is no mechanism to regulate the I/O on a datastore level. As shown in the bottom by the Storage Array Queue in this case VM003 ends up getting more resources than VM001 and VM002 while all of them from a shares perspective were entitled to the exact same amount of resources. Please note that both Device Queue Depth’s are 32, which is the key to Storage I/O Control but I will explain that after the next diagram.

As stated without SIOC there is nothing that regulates the I/O on a datastore level. The next diagram shows the same scenario but with SIOC enabled.

After SIOC has been enabled it will start monitoring the datastore. If the specified latency threshold has been reached (Default: Average I/O Latency of 30MS) for the datastore SIOC will be triggered to take action and to resolve this possible imbalance. SIOC will then limit the amount of I/Os a host can issue. It does this by throttling the host device queue which is shown in the diagram and labeled as “Device Queue Depth”. As can be seen the queue depth of ESX02 is decreased to 16. Note that SIOC will not go below a device queue depth of 4.

Before it will limit the host it will of course need to know what to limit it to. The “datastore-wide disk scheduler” will sum up the disk shares for each of the VMDKs. In the case of ESX01 that is 2000 and in the case of ESX02 it is 1000. Next the  “datastore-wide disk scheduler” will calculate the I/O slot entitlement based on the the host level shares and it will throttle the queue. Now I can hear you think what about the VM will it be throttled at all? Well the VM is controlled by the Host Local Scheduler (also sometimes referred to as SFQ), and resources on a per VM level will be divided by the the Host Local Scheduler based on the VM level shares.

I guess to conclude all there is left to say is: Enable SIOC and benefit from its fairness mechanism…. You can’t afford a single VM flooding your array. SIOC is the foundation of your (virtual) storage architecture, use it!

ref:
PARDA whitepaper
storage i/o control whitepaper
vmworld storage drs session
vmworld storage i/o control session

  • « Go to Previous Page
  • Page 1
  • Interim pages omitted …
  • Page 23
  • Page 24
  • Page 25
  • Page 26
  • Page 27
  • Page 28
  • Go to Next Page »

Primary Sidebar

About the Author

Duncan Epping is a Chief Technologist and Distinguished Engineering Architect at Broadcom. Besides writing on Yellow-Bricks, Duncan is the co-author of the vSAN Deep Dive and the vSphere Clustering Deep Dive book series. Duncan is also the host of the Unexplored Territory Podcast.

Follow Us

  • X
  • Spotify
  • RSS Feed
  • LinkedIn

Recommended Book(s)

Advertisements




Copyright Yellow-Bricks.com © 2025 · Log in