Distributed vSwitches, go Hybrid or go Distributed?

Yesterday I was answering some question in the VMTN Forums when I noticed that someone referred to my article about Hybrid vs full Distributed vSwitch Architectures. This article is almost two years old and definitely in desperate need of a revision. Back in 2009 when Distributed vSwitches where just introduced my conclusion in this discussion was:

If vCenter fails there’s no way to manage your vDS. For me personally this is the main reason why I would most like not recommend running your Service Console/VMkernel portgroups on a dvSwitch. In other words: Hybrid is the way to go…

As with many things my conclusion / opinion was based on my experience with the Distributed vSwitch and I guess you could say it was based on how comfortable I was with the Distributed vSwitch and the hardware that we used at that point in time. Since then much has changed and as such it is time to revise my conclusion.

In many environments today Converged Networks are a reality which basically means less physical NICs but more bandwidth. Less physical NICs results in having less options with regards to the way you architect your environment. Do you really need you management network on a vSwitch? What is the impact of not having it on a vSwitch? I guess it all comes down to what kind of risks you are willing to take, but also how much risk actually is involved. I started rethinking this strategy and came to the conclusion that actually the amount of risk you are taking isn’t as big as we  once all thought it was.

What is the actual issue when running vCenter virtually connected to a Distributed Switch? I can hear many of you repeat the quote from above “there’s no way to manage your vDS”, but think about it for a second… Do you really need to manage your vDS in a scenario where vCenter is down? And if so, wouldn’t you normally want to get your Management Platform up and running first before you start making changes? I know I would. But what if you really really need to make changes to your management network and vCenter isn’t available? (That would be a major corner case scenario by it self but anyway…) Couldn’t you just remove 1 NIC port from your dvSwitch and temporarily create a vSwitch with your Management Network? Yes you can! So what is the impact of that?

I guess it all comes down to what you are comfortable with and a proper operational procedure! But why? Why not just stick to Hybrid? I guess you could, but than again why not benefit from what dvSwitches have to offer? Especially in a converged network environment being able to use dvSwitches will make your life a bit easier from an operational perspective. On top of that you will have that great dvSwitch only Load Based Teaming to your disposal, load balancing without the need to resort to IP-Hash. I guess my conclusion is: Go Distributed… There is no need to be afraid if you understand the impact and risks and mitigate these with solid operational procedures.

Be Sociable, Share!

    Comments

    1. says

      Duncan,

      Couldn’t agree more! My biggest obstacle has been moving vCenter around between clusters on different vDS, and then getting it online long enough to modify the vDS port group by switching a dvUplink to a temp vSwitch. It’s more annoying than anything. Would love to see the dvSwitch move to a VSM model similar to N1KV that would be agnostic to vCenter server.

      Unfortunately, from what I’ve heard, this isn’t going to change anytime soon, so it’s something we need to accept and move on from. I wrote up a quick HOWTO recovery walkthrough the other day here: http://nt-ap.com/eAxzBr

      Would love your feedback!

      @that1guynick

    2. Sketch says

      One issue I see w/ going full dvSwitch (1000v) is that in cases that role separations are enforced, the VMware admin on duty at the time of the outage may not have the knowledge base/skill set to dig into the dvSwitch, where they should know very well how to move around in the VSS fields.

      Outages happen at the most inopportune times, so getting things to a point where they are at least tolerable in a timely fashion is key. I think it depends on the skill set of the duty crew at the time…

    3. says

      Duncan,

      Totally agree with you here. We have the Nexus 1000v vDS here at my company and it’s *so* awesome.

      When I have a new host to add to my cluster, I install ESXi, apply a host profile, and because of the vDS and host profiles, the host is up and running in no time.

      Easy as pie.

      As far as management goes, I can vouch for the fact that it’s not a big deal to have it unavailable when vCenter is unavailable because you really don’t want to muck around with it without vCenter anyway.

      In other words, not a big deal at all. The benefits of the vDS far outweigh the risks.

    4. says

      Couldn’t vCenter heartbeat help minimize the risk of losing you vCenter and therefor minimize the rick of not being able to communicate with your vDS?

      I am in the middle of designing a IaaS-platform based on vSphere and have to take the vSwitch/vDS-decision soon. Before your blogpost I would have gone hybrid – but I will definitely reconsider that.

    5. says

      Funny, while doing my VCAP-DCD exam yesterday this was one of the ‘drawing’ questions and to be honest: I did make a ‘dedicated’ distributed design since so much has been changed. Now you can use commandline to create a temp vSwitch and reconnect certain ports to it.

      But as always: people need to be convinced first. Allow a test environment to prove its operational readiness in a non hybrid design. While technically possible – the ‘feeling’ of control needs to be right too – and if you’re just a GUI kind of admin, be ready to dive into commandline/API.

      Now I’m just hoping the exam rated my design correctly ;-)…

    6. says

      “Couldn’t you just remove 1 NIC port from your dvSwitch and temporarily create a vSwitch with your Management Network? Yes you can!”

      It would be useful to describe how this is accomplished with vCenter/control plane being down.

      “So what is the impact of that?”

      Depending on the disposition of the Management Network and vCenter network, a potential impact is an immediate VLAN change on a production network. Best case scenario, you’re also the network engineer so you can fly this under the radar. Worst case scenario, you work in a larger/siloed organization where this work will be required from the network team and there’s heightened awareness to make the immediate temporary change, then to revert it back when done.

      • says

        I kind of agree with Jason, keep them separated, based on our experience with the Nexus 1000V, for example.
        We did 2 upgrade procedures of the Nexus VSM (13a to 1.4) both failed – we had to reinstall VSM modules, relicense them, do everything from scratch.
        Maybe Nexus product got better over the last year, but still has room for improvement, lots of it.

        As Jason said, to remove an uplink you have to manage the Nexus, which you cannot do if your vcenter (front + DB )has its network down (scenario: VSM crashes, vcenter service stops, host reboots, result: VEM has no configuration info).
        You can probably hack your way through the remote support console, remove the vem, reboot the host, create a new management network….and work your way from there, with bringing up vcenter,at least 1 vsm, etc. etc.

        I agree a vSwitch adds one more thing to manage ( x times how many hosts you got) – but if you got dvS you got host-profiles so that is easy.
        Also you don’t need load balancing on your management traffic, it’s just a few IP addresses.
        In our case (we use UCS + Palo) I do not mind creating 2 additional virtual NICs, and resting easy, that our management traffic is ok.

        Personally be more that happy to use dvSwitches when they are just as reliable as vSwitches (e.g. you don’t need vCenter for some basic management tasks to function).

        I’m not sure how our 1000V experience translates to the VMware dvSwitch, just giving everyone a fair warning before committing their infrastructure to it.

    7. Mike says

      My only question is storage networks – is it safe to do? It’s not like the dvSwitch will close off if my virtual vCenter goes down, making it impossible to see the storage, right?

      • says

        With Nexus 1000V, yes, but in this scenario:

        vcenter goes down (it,or the database)
        VSM module goes down (for whatever reason)
        Host reboots

        So, when the VEM module on the host comes up after the host reboot, it will have no idea whether that traffic is allowed, it needs confirmation from the VSM, and if you VSM is down…bummer

        Fix: Configure the port-profile for your IP storage with “system-vlan option) (system-vlan means, no matter what security is in place on the VSM for that profile, pass traffic, it is kind of a bad practice to use it, except for VSM management tasks.)

    8. redstorm_ says

      Questions: At what point is it better to use vDS versus vSS? How many hosts would constitute vDS as a better option? The reason I ask is that we rarely make that many changes to host network configurations and with Enterprise Plus using Host profiles works perfectly (provided you can put a host into maint mode). vDS is a good option when you have large environments…I just don’t think it’s necessary until you reach a threshold of hosts…just my .02. I have tried it though and it is a nice option to have…

    9. Fred Peterson says

      I look at as the onboard NICs are management traffic only and thus create them as a local vSwitch. The adapters in slots are the ones that go in a vDS.

      I keep IP storage on vSwitches also, again because maintenance can be performed/adjusted directly on the host. Think semi-disaster more then “oops vCenter is down”

      In my environment vmnic1 is for Management Port/Service Console and vmnic2 for vMotion. They are on the same standard vSwitch though.

    10. says

      Don’t forget that the console in 4.1 now has an option to move the management port to a standard vswitch. It works really great in a pinch.

      I still think you need a hybrid design when doing iSCSI multi-pathing. Not 100% sure but I think it’s still not supported on dvs.

    11. Clement says

      Hey there. I never had the chance to implement and really test dvSwitches. I’ve got to question, if you don’t mind taking a few minutes to enlight that dark spot in my VI knowledge that dvSwitchs are :
      – In a fully “dvswitched” virtual infrastructure network, if you connect directly on a ESXi host and access the network configuration tab. What would you see ? Standard vSwitches ?
      – What if you loose your vCenter database ?

      Thanks.

      Clement.

      • says

        Clement, to answer you:

        1. You can see the dvSwitch config, but can’t modify it.
        2. It’ll be time to rebuild your dvSwitch config if you can’t restore from backups (I assume that’s what you mean by “lose” the database). If you mean the DB drops offline, then you just can’t modify the configuration of the dvSwitch until vCenter is operational again.

        Hope that helps,

        Dan Hayward

        • says

          I have to say, I run hybrid, and I’m glad of it.

          We’re running ESX 4.1, so moving back to a vSS isn’t as simple as far as I’m aware if there is a problem… (The ESXi migration will be underway shortly).

          We recently had an issue with a customer’s vCenter DB server where the DB corrupted, and the backups were all skewed too, so we lost the database completely. Having management networks on VSS made life easier, as we could add the hosts back into a fresh vCenter, and then reconfigure a new vDS for the VM’s.

          I’d have to say, if I had 10GbE NIC’s then I’d put them onto the vDS and the 1GBps onboard NICs into a VSS for management traffic, I’d just feel safer that way too…

          And of course, you can always have a second management network that’s not on the vDS in case of some major vCenter issue… Just an idea… :-)

    12. says

      To me the biggest driver for Distributed vSwitches is 10GbE. With 1GbE you use quite a few nics to separate VMotion, IP Storage and FT. Management rarely have special bandwidth requirements, but certain VM networks may also have special needs, and may need an extra pair of nics or two. With 10GbE you normally use only 2 physical nics and you can solve this issue by using vDS with LBT and NetIOC.

      Lars

    13. says

      Clement,
      If you lose your vCenter database you will loose the ability to configure networking of your VMs, but not their network connectivity. In the ESXi 4.1’s menu you can now convert the hosts distributed switches into traditional vSwitches.

      Lars

    14. Scott Bethke says

      After several deployments now with the Cisco vDS (Nexus 1000v) in both multi-phy NIC and dual 10GBE CNA/PALO I can safely say that until the cisco product matures I will not run management network on the 1000v. The reasons end up being fairly simple. So I’ve gone through all of the most recent code revisions and upgrades, bumping into bugs all along the way (Welcome vGuys to what we used to affectionately call “IOS Roulette”) We noticed in certain situations that the VEM would go into a Zombie state. It would still forward packets for all of the veth’s it knew about at the time of zombie instantiation but new hosts that happen to move over were DOA.. and it was contagious meaning many if not all VEM’s started to behave this way after the first instance. So what do I do? vMotion right? Reboot? Well no, I cant vMotion because it brings the hosts down once they arrive at the new zombie VEM (these are production environments).. Its simple, I get on the ESXi host and simply stop then start the VEM (quickly).. However, as soon as I stop the VEM.. there goes my management network. So sneakernet ends up being your course of action. In a production environment I don’t have time to go to the server or fiddle with JAVA versions while trying to coats a correct thinapp’ed version of IE to allow the remote console window to not only open but also work.

      Now restarting VEM’s could probably be fixed by them simply adding a restart option.. But the fact that a bug like this could happen is reason enough to keep management separate..

      My top two reasons for running the good ole vss for Management is 1) restarting VEM’s.. it happens. and 2) NX-OS has roots in IOS, and is still managed (code wise) just like IOS was back in 1989 thus Out of band needs to plan for Out of IOS.. or NX-OS in this case.

      Standard vDS appears to be much more stable than Nexus so I would probably be ok (until proven guilty) that management is fine on the normal vDS.

      • Mike Rizzo says

        Scott,

        I am in the middle of planning a Nexus 1000V deployment and your description of a “Zombie” VEM situation concerns me. Could you elaborate a bit more about what you observe when this situation occurs, and if you have any information from Cisco about the problem or a solution to it? Since many of our VM’s run mission-critical applications our concern goes beyond loss of management connectivity.

        Any details on your experience would be much appreciated.

        • Scott Bethke says

          Mike,

          Just caught this sorry it’s late.. the zombie state was a known bug as it turned out in SV1 to 3 of 4.0.4 if I remember my numbers correctly. We have moved on to 4.2 and do not see that issue any more. However feature growth has been slow.. so perhaps they are trying to limit new features before extensive testing. With the 1000v you are running basically a normal vDS with lower configuration maximums, and added complication of an NX-OS CLI on the config front, and a pair of HA VSM’s to handle things like distributed configuration to all of the VEM’s. I hope they are concentrating on making that “complication” aspect more of a “must have” .. as VCE and FlexPod both seem to require 1000v as part of the stack. While the latest release seems to be very stable, I still must recommend that anyone doing Nexus 1000v go with a hybrid setup.

    15. says

      DVSwitch for VM traffic should be fine. If you plan to run ISCSI or NFS for the shared storage, I think it should be run on dedicated physical link for ease of management and keep it running on standard vswitch. Just my 2 cents

    16. Mihai says

      We run vSS all the way here, we don’t want to depend on vCenter at all for basic virtualization functionality.

      When we add a new host we just use a script that copies all configuration from an existing host, no work.

    17. pc says

      I would have to lean towards a hybrid design as well if forced to choose vds. In my opinion, vds really only comes into play when using 1kv. How much time does it really save you to be able to add a host to a vds and have settings pushed down? A carefully crafted powercli script can be just as effective and comparable in execution time. Honestly, when i factor in the amount of time it takes me to play “find the vcenter server” and “steal a nic” in the event of a power outage it makes vds much less attractive.

      That brings me to my hybrid choice backed by the introduction of the Palo. Best of both worlds.

    18. tom miller says

      Little late with a reply but needed to think about this one. Curious how folks are handling moving from vCenter 4 to 4.1 when your hw platform was 32 bit and your using vDS? Typically we just build a new vCenter on 64 bit hw or make it a VM running on a 64 bit OS. However, since vCenter is different from the original vCenter and the vDS control plane is also different – how would you reconnect the old I/O plane on the host to the new and different vCenter control plane?

    19. Jo in NYC says

      Kind of struggling with this decision on HP FLEX-10 blades. In an environment where only I have a deep understanding of dvswitches, and where the vcenter database is a total “blackbox” managed in another silo, seems like a potential for disaster. Not comfortable enough with powercli scripting to even make it a moot point.

      I also worry when its time to upgrade the vcenter database from 4.1 to 5.0. With the potential to bring down the whole ESX environment, I may end up taking the migration over in-place route. If only NIOC wasn’t so useful in a blade environment….

    20. James Hess says

      I’m thinking straight up complete vDS across the board, all physical NICs; with Ephemeral binding for critical management, vCenter, and storage networks.

      The idea is that since Ephemeral port groups behave a lot like standard vSwitches without having to go to a hybrid config… since new VMs can be created or plugged into those with vCenter down; it seems like a safety blanket.

      This can also be combined with a script-maintained ‘backup vSwitch’ attached to no physical NICs prepared with port groups / VLANs mirroring the dvSwitch configuration with VSS-* in front of each port group name.

      The idea being… if vCenter needs to be rebuilt…
      plug those NICs into the standard vSwitch and use a script to migrate all the VMs over to it, until the new dVS can be prepared.

    21. Jorge says

      Hi Tom Miller- We are planning to migrate from vCenter 4.0 (32bit) to vCenter 4.1 (64bit) and have a number of dvSwitches. Did you ever perform your Migration?
      If you did, how did you reconnect the old I/O plane on your hosts to the new and different vCenter control plane? Where you able to migrate without downtime?

      Has anyone out there performed a 4.0 to 4.1 migration that included dvSwitches? And if so, what issues occurred? Are there any special precautions that have to be taken?

    22. Pooriya says

      Hi,

      I am about to deign a new virtual infrastracture network and I am really confused in choosing between vNetwork Distributed Switch and vSwitch. I wonder if I should remove the hosts and their VMs from normal vSwitches and connect the them to the vDS. or if I should leave the hosts and their VMs on the normal vSwitches and still connect them to vDS. How can I remediate from vCenter failure if I use vDS please? Thanks a lot.

    23. jimmy says

      Our standard config is running dual 1Gbps from LOM NIC’s on a standard vswitch for mgmt. only, then install dual port 10GbE HBA NIC’s which we setup as a dvSwitch, then VLAN off for mgmt., vmotion, prod, dev and iSCSI. It keeps mgmt separate and converges all other networks nicely over high speed link, whilst still gaining 99% of benefits of dvSwitch.

    24. says

      @pooriya
      vDS is saved in both vCenter and Host. One copy in the vCenter database.You can also find the same copy in the local host at /etc/vmware/dvsdata.db. This local cache will be updated by vCenter every 5 minutes.

      Regards
      Lingeswaran
      http://www.unixArena.com

    Leave a Reply