As not many people realize this lets repeat it… NIOC controls traffic on a NIC port level. Not on a NIC pair, not on a host level and not on a cluster wide level. On a NIC port level!

SIOC does IO control on a Datastore-VM layer. Meaning that when a certain threshold is reached it will determine on a datastore wide level which hosts and essentially which VMs get a specific chunk of the resources. SIOC prevents a single VM from claiming all IO resources for a datastore in a cluster. SIOC is cluster wide on a datastore level! It basically is the police officer who asks your neighbor to tone it down when as he is bothering the rest of the street.

Yes, enabling SIOC and NIOC together makes a lot of sense!

If the inability to quickly provision a new VM or to reconnect a vNIC while vCenter Server is unavailable has kept you from considering a pure vDS network architecture, ephemeral port groups may be a suitable safety net.  You would not even need to use ephemeral port groups for production virtual networks — simply create a few to have as backups for accessing the most critical VLANs.

This started a discussion internally as the default setting is not Ephemeral but Static. So the question that this resulted in was should we define a new standard or are the “Static” port binding just as good as Ephemeral? I believe that many people are hesitant of using a pure vDS infrastructure due to the inability to make changes to the vDS when vCenter would be unavailable. This applies to both ephemeral and static however and actually leads to another point, which we won’t discuss now, vCenter resiliency. Now, from a virtual machine perspective even if vCenter is down, and Static is used as the port bindings, the virtual machine can be powered on and off. With Static all ports are pre-defined on the host level and when a virtual machine is assigned a port it can consume it. Now the difference between Ephemeral and Static is that Ephemeral allows you to assign “new ports” to new virtual nics or virtual machines. I guess the question is how often do you make changes to the network of your virtual machines when vCenter is down and what type of changes?

Seriously, do we really want to make substantial changes to our environment when our management platform is not available? I believe we shouldn’t and I also feel that Static portgroups are the way forward, they have more or less the same level of flexibility Ephemeral have and on top of that Static offers a lot of advantages from a scaling perspective!

Source: VMware vSphere 4.1 Networking Performance

Download:
http://www.vmware.com/files/pdf/techpaper/Performance-Networking-vSphere4-1-WP.pdf

Description

This paper demonstrates that vSphere 4.1 is capable of meeting the performance demands of today’s thoughput-intensive networking applications. The paper presents the results of experiments that used standard benchmarks to measure the networking performance of different operating systems in various configurations. These experiments examine the performance of VMs by looking at VMs that are communicating with external hosts and are communicating among each other, demonstrate how varying the number of vCPUs and vNICs per VM influences performance, and show how the scalability results of overcommitting the number of physical cores on a system by adding four 1-vCPU VMs for every core.

If vCenter fails there’s no way to manage your vDS. For me personally this is the main reason why I would most like not recommend running your Service Console/VMkernel portgroups on a dvSwitch. In other words: Hybrid is the way to go…

As with many things my conclusion / opinion was based on my experience with the Distributed vSwitch and I guess you could say it was based on how comfortable I was with the Distributed vSwitch and the hardware that we used at that point in time. Since then much has changed and as such it is time to revise my conclusion.

In many environments today Converged Networks are a reality which basically means less physical NICs but more bandwidth. Less physical NICs results in having less options with regards to the way you architect your environment. Do you really need you management network on a vSwitch? What is the impact of not having it on a vSwitch? I guess it all comes down to what kind of risks you are willing to take, but also how much risk actually is involved. I started rethinking this strategy and came to the conclusion that actually the amount of risk you are taking isn’t as big as we  once all thought it was.

What is the actual issue when running vCenter virtually connected to a Distributed Switch? I can hear many of you repeat the quote from above “there’s no way to manage your vDS”, but think about it for a second… Do you really need to manage your vDS in a scenario where vCenter is down? And if so, wouldn’t you normally want to get your Management Platform up and running first before you start making changes? I know I would. But what if you really really need to make changes to your management network and vCenter isn’t available? (That would be a major corner case scenario by it self but anyway…) Couldn’t you just remove 1 NIC port from your dvSwitch and temporarily create a vSwitch with your Management Network? Yes you can! So what is the impact of that?

I guess it all comes down to what you are comfortable with and a proper operational procedure! But why? Why not just stick to Hybrid? I guess you could, but than again why not benefit from what dvSwitches have to offer? Especially in a converged network environment being able to use dvSwitches will make your life a bit easier from an operational perspective. On top of that you will have that great dvSwitch only Load Based Teaming to your disposal, load balancing without the need to resort to IP-Hash. I guess my conclusion is: Go Distributed… There is no need to be afraid if you understand the impact and risks and mitigate these with solid operational procedures.

In our book we give two examples, one of which is the simple version with a single “Service Console Network” and one with a dual “Service Console Network” configuration. Now I figured I could update both but I’d rather do just one and explain why I prefer to use this one. The one that I have picked is the single “Management Network” setup. The main reason for it being is the reduced complexity that it brings and on top of that multiple Management Networks will make sense in an environment where you have many NICs and Switches but with all these converged architectures flying around it doesn’t really make sense anymore to have 4 virtual links when you only have 2 physical. Yes I understand that something can happen to a subnet as well, but if that is the case you have far bigger problems than your HA heartbeat network failing. Another thing to keep in mind is that you can also mitigate some of the risks of running into a false positive by selected a different “Isolation Response”, typically we see these set to “Leave Powered On”.

The below is an excerpt from the book.

Although there are many configurations possible and supported we recommend a simple but highly resilient configuration. We have included the vMotion (VMkernel) network in our example as combining the Management Network and the vMotion network on a single vSwitch is the most commonly used configuration and an industry accepted best practice.

Requirements:

• 2 physical NICs
• VLAN trunking

Recommended:

• 2 physical switches

The vSwitch should be configured as follows:

• vSwitch0: 2 Physical NICs (vmnic0 and vmnic1)
  • When multiple physical PCI devices are available make sure to use a port of each to increase resiliency
• 2 Portgroups (Management Network and vMotion VMkernel)
• Management Network active on vmnic0 and standby on vmnic1
• vMotion VMkernel active on vmnic1 and standby on vmnic0
• Failback set to No

Each portgroup has a VLAN ID assigned and runs dedicated on its own physical NIC; only in the case of a failure it is switched over to the standby NIC. We highly recommend setting failback to “No” to avoid chances of a false positive which can occur when a physical switch routes no traffic during boot but the ports are reported as “up”. (NIC Teaming Tab)

Pros: Only 2 NICs in total are needed for the Management Network and vMotion VMkernel, especially useful in Blade environments. This setup is also less complex.

Cons: Just a single active path for heartbeats.

The following diagram depicts the active/standby scenario:

To increase resiliency we also recommend implementing the following Advanced Settings where the ip-address for “das.isolationaddress” should be a “pingable” device which is reachable by the ESXi hosts, preferably on the same subnet with as little hops as possible:

 das.isolationaddress = <ip-address>
 das.failuredetectiontime = 20000

vApp fenced to an Internal Org Network

Use case:

1. Environments where vApps are copied and redeployed for  test and development purposes. There is no connection back to the customers datacenter to avoid any interference that could be cause by these test environments.

We will start with the basics. The flow of the network in this case will be:

Although only two different type of networks are used, this could of course result in multiple layers if and when for instance multiple vApp Networks are created. For the purpose of this exercise we will create a vApp with 3 VMs including two different networks. You could say you classical three-tier application.

1. WEB = Web Frontend
2. APP =Application Server
3. DB = Databaser Server

As you can imagine we don’t need users accessing the Application or Database Server so these two will be on a separate network segment. The Web Frontend will need to be accessible though and it will need to be able to access both the Application and the Database Server. Logically speaking that will look as follows:

Please note that the Org Network doesn’t connect back to anything! This means that in order for you to connect to your WEB vm you will need to go through the vCD Portal! Of course you could still test if your web services are working by simply deploying a desktop VM with windows XP in the same Org. Now I can hear some of you think why not just a NAT-Routed Org Network, well that is something that would work as well, but than it would be really similar to what use case 1 provided and this is purely for educational purposes.

Creating the Networks

The first step of course is to ensure you have a Network Pool. If you haven’t already created, you can use Part 2 of this series as a reference.  I am assuming here you already have a network pool and will go straight to the Org Network, which is option 7 on the home screen.

Now you will need to select the Org that this Network will belong to and then you can decide what type of network you will create. You can do this in either “Typical” or “Advanced” mode. Both will give you the same options but it is named slightly different and Advanced will only allow you to create 1 network at a time where with Typical you can create multiple. As we have used Typical in the previous use case we will use Advanced this time. We are going to create a fully isolated Org Network so we will select “Internal Organization Network”.

Next up we will need to select a network pool. Now you might ask yourself why we will need one when the Org Network is completely isolated? Well we will need cross-host communication when vApp/VMs need to communicate with each other and don’t reside on the same host. Although it sounds very logical, it is often overseen that this is what a network pool does. It enables cross-host communication. In this case we will select the vCloud Network Isolation Network Pool.

Now we will need to specify the IP details for this Org Network. These IP addresses will be consumed by the VMs that are configured to use the “static pool”, in our case that will be the vShield Edge device that is deployed as part of this Isolated Network (deployed for DHCP services etc) and the WEB virtual machine.

Of course we will need to give it a name. I tend to use the name of the Org and the type of Org Network I created.

Now we will need to build a vApp. As stated this vApp will contain multiple VMs.

We will give it a name.

And we will start adding multiple VMs to it. The WEB virtual machine will have 2 NICs as it will need to connect to a device outside of vApp and to two VMs inside of the vApp.

The following two VMs “APP” and “DB” will be configured with a single NIC as they will only need to connect to each other, all contained within the vApp.

Now this is the part where we will assign specific network segments to the NICs. For WEB we will connect “NIC 0″ to the Internal Org Network and NIC 1 will need to be connected to a vApp Network.

This vApp Network however will need to be created first. Please note that this is a vApp network, so only available to those VMs which are part of this vApp! Again we will need to specify IP details for the VMs to consume.

When we have done this and have given the vApp network a name we can connect the remaining VMs to the same network.

Now in order to have multiple copies of the same vApp running within the Org we will select “Fenced” mode for the vApp which basically will deploy a vShield Edge device.

I guess this diagram that vCD creates makes it a bit more clear what your vApp connectivity will look like:

And if that isn’t enough you can also check the Maps functionality that vCenter offers. This will give you a great view of how this vApp is connected within vSphere.

So what about that desktop? And what about if we have two copies of that vApp running? Well this is what the map would look like if when we have created these. On the middle left you will see the desktop that is used for testing the WEB VMs. Both WEB virtual machines can be accessed through the VSE device, which of course means that you will need to setup NAT, but we will leave an in-depth explanation around that for the next article.

Summary

vCloud Director Network is really powerful, but as shown by this use case it can get very complex rather fast especially when you are using multiple layers. In this example we kept it simple by using an isolated network, an External NAT/Routed Org Network would have added another layer. Features like vCenter Maps will however make it easier to understand what has been created on the vSphere layer to enable these layers of networking, make sure you take advantage of functionality like this when exploring vCD!

vApp directly connected to an External routed Org Network

Use cases:

1. Internet connection for the VMs in your virtual datacenter. Firewall can be enabled to block all incoming traffic.
2. Publicly publishing a single “service” externally by enabling NAT on the vShield Edge device. In this case all incoming traffic will be blocked and only a single IP will be translated and route back to that particular VM.

We will start with the basics. The flow of the network in this case will be:

As explained in Part 1 the External Network is backed by a Portgroup. This portgroup can be a regular portgroup on a vSwitch, or one on a dvSwitch or even the Nexus 1Kv. We will start by creating a dvPortgroup.

External Network

Lets first create a dvPortgroup within vCenter. This is the dvPortgroup that the External Network will use. We will give it a VLAN ID for layer 2 isolation. In this case we use VLAN ID 105 and label the dvPortgroup as “dvExternal-105″.

Now we will need to create a network within vCD that enables your vApp and Organization to use this dvPortgroup we just created. We do this by creating an “External Network”. (option 3 on your home screen in vCD.) First we will need to select the correct dvPortgroup we created:

Next thing to do is specify the associated IP Range, Gateway, Netmask etc. The IP-Range is used for any VMs that are directly connected to this External Network and for the vShield Edge devices. But we will show that later in this article.

Next up is is giving the External Network a name, we will keep it simple and name it “External – vlan 105″:

That is it for the External Network part. Now lets create an Org Network.

Org Network

We will create an External Org Network which is routed to an External Network. (On your home screen go to “7 Add another network to an organization”.) Select the Organization it needs to connect to first and then the real magic starts.

We will use the typical setup. We have unticked “Create an internal network”, and we have selected “Routed connection”:

The cool thing about the network section of vCD is that is shows you what it is building. In this case you can see that the vApp is directly connected to the External Org Network (NAT-Routed) which in its turn is connected to the External Network through a vShield Edge device. The next step is to select the correct External Network that this External Org Network connects to:

Please note that we also have selected a network pool, in this case the vCloud Network Isolated Pool! Next we will need to specify the associated IP Range, Gateway, Netmask for the Org Network. Now you might think that we have already done this but that was for the External Network! The pool of addresses will be used for any device that sits within the Org Network boundaries.

Of course the final step is giving this Org Network a name:

vApp layer

As this post is about networking I will skip the creation of the vApp itself but will show you what we have done in a single screenshot. As this screenshot below shows the VM is directly connected to the Org Network labeled “YB-NAT-Ext-Org”:

Connecting the dots

Now that we have shown you how this is created within vCD you would probably want to know what this results in on a vSphere layer. When we created the Org Network a dvPortgroup was automatically created. This automatic creating was enabled by the use of a network pool. The network pool in this case was a vCloud Network Isolation backed network pool.

The screenshot below shows the dvPortgroup that represents the Org Network. The VM that was created called “Direct”, however vCD uses IDs to uniquely identify VMs and as such it is labeled as “1227504509-Direct” within vSphere. Please note the “F46″ in the name of the dvPortgroup. This means that it is using a fenced network with ID 46. (fenced –> vCloud Network Isolation) This Network Pool happens to use VLAN 107 (V107), which was defined when the pool was created and is also shown in the screenshot below.

In order for VM “1227504509-Direct” to communicate to the outside world it will need a connection to the External Network. As shown and described above VMware vCloud Director uses vShield Edge to do this. In other words, the vShield Edge device will have multiple NICs. This is shown in the following screenshot. The External Network portgroup contains a vShield Edge device (vse-651240915) which is the same device as shown in the screenshot above.

This is the vShield Edge device that enables the VM “1227504509-Direct” to communicate with the outside world, as it is connected to both portgroups.

Traffic Flow

As it took me a while to understand how this worked, I have created a couple of diagrams. The first diagram shows all components we created and how they are linked:

I guess this is still not saying much. Lets add the flow of the traffic to this diagram by extending it with another vApp. What if you would have two vApps connected to the same Org Network and both VMs of these vApps are on a different host in your cluster and the first VM wants to connect to the second VM? What does the flow of traffic look like? As you can see in the diagram below the VM of the first vApp is connected to the same dvPortgroup. However as both vApps reside on a different host the traffic will need to go to the physical switch layer first:

The other scenario I wanted to show is where a vApp wants to connect to a device on the outside world. In this case I labeled it as “internet” but it could be anything. Also I have assumed that the vShield Edge device resides on a different host than the VM that wants to connect to the internet.

It took me a while to write this “use case”. I hope this makes vCD networking slightly better to understand… but again the key here is to play around with it. If there are any questions please don’t hesitate to reach out to me! If I can find the time I will write another “use case” or maybe I will ask some of the other guys in my team to do something similar.

Network Pools

As described in one of my earlier articles vCD mainly revolves around pooling of resources. This also goes for networking and more specifically for the Org Network and the vApp Network. Each vApp and Org Network will consume a network (segment) out of a defined pool. This network pool typically isolates network “segments” on layer 2 from the other networks in the pool. There are currently 3 types of network pools:

1. VLAN Backed
2. vCloud Network Isolation Backed (VCNI)
3. Portgroup Backed

Each of these pools have specific requirements, recommendations and constraints and they will be listed below. However all of the three different types of pools are interchangeable; with that meaning a segment from a VLAN-Backed Pool and a Portgroup-Backed pool each offer the same functionality to your Org and vApp network and basically should be seen as a means of intra-Cloud transportation. Or to simplify it even further, it provides a wire. This wire enables VMs in a vApp to communicate to each other or vApps in an Org to communicate to each other. This also means, and that might sound very logical to some, the distributed vSwitch / Nexus / vSwitch needs physical uplinks in order to enable communication across multiple hosts in a cluster!

When is a segment of the network pool used? The answer is simple, whenever you create a vApp or Org Network which is not directly connected to the above layer a segment of your designated network pool will be used. Before you ask, the reason the network pool is not used in the case of a “directly” connected Org or vApp network is because the Org/vApp network is just a logical object in that case and the VM/vApp ends up directly in the portgroup the layer above. I will show you what that looks like in Part 3 of this series which will tie part 1 and 2 together including vShield Edge.

Now lets assume you created a VLAN-backed or a vCloud Network Isolation-backed pool. When an isolated or NAT routed vApp/Org network is created vCD will automatically instantiate a new network segment. This segment is essentially a portgroup on a distributed vSwitch. This portgroup will provide layer two isolation between the different segments. As a distributed vSwitch is used every host in your cluster will instantly have the same portgroups available. There is a very specific reason I only mentioned the VLAN Backed and the VCNI Backed pool, for the Portgroup Backed pool you will need to manually pre-create all portgroups and of course ensure all those portgroups are available on every host in the cluster.

To visualize it a bit more I took three screenshots. The first one shows the network pool. The pool is VLAN backed and holds VLAN 1000 through 1100 and these networks will be created on the distributed vSwitched labeled as “dvSwitch”:

Now that we have created a network pool we can create an isolated Org Network, in this case we create a fully isolated Org Network. This isolated org Network aka “Internal network” enables the vApps within your virtual datacenter to communicate to each other:

In order to be able to communicate we need a network segment. This network segment is provisioned from the Network Pool we created earlier, which was VLAN Backed.

After you fill out some of the other details and click “finish” a portgroup is automatically created within vSphere. Although difficult to read the portgroup name contains the Org Network and the VLAN ID. (VLAN 1008 and Org Network “YB-ISO-ORG”). You can even see the task at the bottom of the screenshot:

Hopefully that visualizes the process a bit more. I dropped some steps for the creation of the pool and the Org Network to keep it relatively simple. You can find a full detailed video here. Now as stated earlier each of the the three types of pools have very specific requirements and constraints I listed them below so you can make a decision based on the requirements or constraints you might have.

VLAN Backed

VLAN-backed network pools require availability of a set of unused VLANs. When an Org or vApp network is created which is based on a VLAN-backed network pool a portgroup is created on a dvSwitch and a VLAN is assigned to this portgroup. It should be noted that all VLANs specified for the pool will need to be trunked to the host and that in most environment the amount of available VLANs is limited.

• Requirements: Distributed vSwitch, pool of available VLANs, Physical uplinks need to support VLAN Trunks.
• Recommendations: n/a
• Constraints: Nexus 1000v and regular vSwitches are not supported currently, amount of available VLANs in most environments.

vCloud Network Isolation Backed

vCloud Network Isolation-backed(VCNI) network pools are flexible, easy to configure and do not require VLANs. vCNI provides layer 2 isolation by utilizing a network overlay. This network overlay is provided by MAC in MAC encapsulation and requires a vDS. For each consumed network vCloud Director creates a portgroup and assigns this portgroup a network ID number. This network ID number is used for the encapsulation of the traffic. As explained vCD uses MAC in MAC for the encapsulation of traffic. However, because of that the use of VCNI requires an increase in the MTU of the underlying transport network(dvSwitch) to avoid frame fragmentation due to the minor overhead cause by the MAC in MAC encapsulation.

Now this is a very thin explanation, but I don’t want to go to deep into this as it would only complicate things.

• Requirements: Distributed vSwitch
• Recommendations: minimum of 1 VLAN, MTU Increase (24Bytes, 1500 –> 1524).
• Constraints: Nexus 1000v and regular vSwitches are not supported currently.

Portgroup Backed

Portgroup-backed pools require pre-created portgroups within the vSphere environment. Portgroup-backed pools are in my opinion the least flexible of all three options. However, Portgroup-backed pools do not require vDS and can be based on vSS, vDS or Cisco Nexus 1000v which can be a specific customer or even platform requirement.

• Requirements: All portgroups need to be pre-created and available on all hosts of your cluster
• Recommendations: Use a scripted solution or host profiles to create the portgroups to ensure consistency
• Constraints: n/a

Wrapping up

Hopefully this clarifies network pools a bit. As explained, when creating and Org of vApp network which is not directly connected to the layer above a network of your Network Pool will be used to provide intra-cloud communication. For those who haven’t used vCD at all, believe me it took me a while to grasp these concepts so don’t be shy and ask questions/comment if anything is unclear.

If there are any questions don’t hesitate to leave a comment. Please note that I am deliberately trying to simplify things in this first article as I don’t want you to get lost in any of the layers of networking vCD offers.

Layered

Networking within vCD is built up out of 3 distinct layers.

1. External Network
2. Org Network
3. vApp Network

These three layers have been created to give the end-user the flexibility needed in a multi purpose virtual datacenter. I have depicted all three layers in the following diagram which shows the logical relationship between the layers:

Some of you technical guys might say, that’s nice but I would like to see something less abstract. I created the following diagram which depicts the different layers in a different way. The diagram shows the three layers. I created a single External Network which links to two Org Networks. These Org Networks in its turn connect to multiple VMs(Org Y) and multiple vApps(Org X).

This is just an example however that illustrates possible network connections and as can clearly be seen it can be rather complex. As demonstrated there are multiple ways to connect vApps to each other or the outside world.

Now that we know some of the basics I will break down the three layers of networking. The  first before we will discuss any of the advanced options like vShield Edge or network pools

External Network

The External Network is used for inter-Cloud connections. Or as I like to call it “your connection to the outside world”. It is the first network “object” that you create within vCD. An External Network is always backed by a Portgroup, meaning that a portgroup needs to exist within vSphere before you can create this vCD network object. This portgroup can be on a regular vSwitch, a dvSwitch or you could use Nexus 1KV. It all works, and all of them are supported!

Of course it is heavily recommended to set this portgroup up with a VLAN for layer 2 isolation, again note that this is an outbound facing connection for your Org or for multiple Orgs.

Examples of External Networks are:

• VPN to customer site
• Internet connection

As said, an external network can be shared between organizations but is typically created per organization and is your connection from or to your virtual datacenter.

I would to stress that, the external network is your exit from your virtual datacenter or your entrance!

Org Network

The second object that is created is the Org Network. The Org Network is used for intra-Cloud connections. Or as I like to call it “Cloud internal traffic”.  An Org Network is linked to an organization and can be:

• Directly connected to an External Network
• NAT/Routed connected to an External Network
• Completely Isolated

With that meaning that although an Org Network is primarily intended for internal traffic, it can be linked to an External Network to create an entry to or exit from your virtual datacenter. Note that it doesn’t necessarily need to be connected to an External Network, it could be completed isolated and used for “Cloud internal traffic” only! A use case for this would be for instance a test/dev environment where vApps will need to communicate with each other but not with the tenants back-end.

It should also be noted that the Org Network is mandatory! Every organization needs an Org Network, it is the only mandatory network object.

Just for completeness, an Org Network consumes a segment from a Network Pool when it is NAT/Routed or Isolated. A network pool is a collection of L2 networks which will be automatically consumed by vCD when needed, and what I call a segment is one of those L2 networks… basically a portgroup. I will explain Network Pools more in-depth in part 2.

When an Org Network is directly connected it is just a logical entity and physically doesn’t exist. Again in one of the following articles(part 3) I will explain what that looks like in vCenter.

vApp Network

The vApp Network kind of resembles the Org Network as it also consumes a segment from a Network Pool. The vApp Network enables you to have a vApp internal network, this could be useful for isolating specific VMs of a vApp. The vApp Network can be:

• Directly connected to an Org Network
• NAT/Routed to an Org Network
• Completely Isolated

It should be noted that the “directly connected” option for both the Org Network and the vApp Network is just a logical connection. In the back-end it will be directly connected to the layer above.

As shown in an earlier diagram and explained above a vApp can contain multiple networks. This can be used to isolate specific VMs from the outside world. An example is shown in the following diagram where only the Web Server has a connection to the Org Network and the App and Database servers are isolated but do connect to the Web server.

Summary

vCD has three different layers of networking. Each of these layers has a specific purpose. The External Network is your connection to the outside world, the Org Network is linked to a specific Organization and the vApp network only resides within a vApp.

That is it for Part 1. Part 2 will focus on the Network Pools and Part 3 will focus on what these vApp, Org and External Networks look like on a vSphere layer and some general best practices.

My tip of the day, if you want to get to know vCD really well, check vCenter every time you make a change and see what happens!

UPDATE: for a full schematic overview check Hany’s awesome diagram.

Scenario:

• Two NICs
• 1 Etherchannel of 2 links
• Both Management and VMkernel traffic on the same switch

I created a simple diagram to depict this:

In the above scenario each “portgroup” is configured in an active/standby scenario. So let’s take the Service Console. It has VMNIC0 as active and VMNIC1 as standby. The physical switch however is configured with both NICs active in a single channel.

Based on the algorithm that etherchannels use either of the two VMNICs will accept inbound traffic. The Service Console however will only send traffic outbound via VMNIC0. Even worse, the Service Console isn’t actively listening to VMNIC1 for incoming traffic as it was placed in Standby mode. Standby mode means that it will only be used when VMNIC0 fails. In other words your physical switch will think it can use VMNIC1 for you Service Console but your Service Console will not see the traffic coming in on VMNIC1 as it is configured in Standby mode on the vSwitch. Or to quote from Frank’s article…

it will sit in a corner, lonely and depressed, wondering why nobody calls it anymore.