Yesterday I got an email about configuring VXLAN. I was in the middle of re-doing my lab so I figured this would be a nice exercise. First I downloaded vShield Manager and migrated from regular virtual switches to a Distributed Switch environment. I am not going to go in to any depth around how to do this, this is fairly straight forward. Just right click the Distributed Switch and select “Add and Manage Hosts” and follow the steps. If you wondering what the use-case for VXLAN would be I recommend reading Massimo’s post.
VXLAN is an overlay technique and encapsulates layer 2 in layer 3. If you want to know how this works technically you can find the specs here. I wanted to create a virtual wire in my cluster. Just assume this is a large environment, I have many clusters and many virtual machines. In order to provide some form of isolation I would need to create a lot of VLANs and make sure these are all plumbed to the respective hosts… As you can imagine, it is not as flexible as one would hope. In order to solve this problem VMware (and partners) introduced VXLAN. VXLAN enables you to create a virtual network, aka a virtual wire. This virtual wire is a layer 2 segment and while the hosts might be in different networks the VMs can still belong to the same.
I deployed the vShield virtual appliance as this is a requirement for using VXLAN. After deploying it you will need to configure the network. This is fairly simple:
- Login to the console of the vShield Manager (admin / default)
- type “enable” (password is “default”)
- type “setup” and provide all the required details
- log out
Now the vShield Manager virtual appliance is configured and you can go to “https://<ip addres of vsm>/. You can login using admin / default. Now you will need to link this vShield Manager to vCenter Server:
- Click “Settings & Reports” in the left pane
- Now you should be on the “Configuration” tab in the “General” section
- Click edit on the “vCenter Server” section and fill out the details (ip or hostname / username / password)
Now you should see some new shiny bright objects in the left pane when you start unfolding:
Now lets get VXLAN’ing
- Click your “datacenter object” (in my case that is “Cork”)
- Click the “Network virtualization” tab
- Click “Preparation” –> “Connectivity“
- Click “Edit” and tick your “cluster(s)” and click “Next“
- I changed the teaming policy to “failover” as I have no port channels configured on my physical switches, depending on your infra make the changes required and click “finish“
An agent will now be installed on the hosts in your cluster. This is a “vib” package that handles VXLAN traffic and a new vmknic is created. This vmknic is created with DHCP enabled, if needed in your environment you can change this to a static address. Lets continue with finalizing the preparation.
- Click “Segment ID“
- Enter a pool of Segment IDs, note that if you have multiple vSMs this will need to be unique as a segment ID will be assigned to a virtual wire and you don’t want virtual wires with the same ID. I used “5000 – 5900”
- Fill out the “Multicast address range“, I used 225.1.1.1-225.1.4.254
Now that we have prepped the host we can begin creating a virtual wire. First we will create a network scope, the scope is the boundary of your virtual network. If you have 5 clusters and want them to have access to the same virtual wires you will need to make them part of the same network scope
- Click “network scopes“
- Click the “green plus” symbol to “add a network scope“
- Give the scope a name and select the clusters you want to add to this network scope
- Click “OK“
Now that we have defined our virtual network boundaries aka “network scope” we can create a virtual wire. The virtual wire is what it is all about, a “layer 2” segment.
- Click “networks“
- Click the “green plus” symbol to “create a VXLAN network“
- Give it a name
- Select the “network scope“
In the example below you see two virtual wires…
Now you have created a new virtual wire aka VXLAN network. You can add virtual machines to it by simply selecting the network in the NIC config section. The question of course remains, how do you get in / out of the network? You will need a vShield Edge device. So lets add one…
- Click “Edges“
- Click the “green plus” symbol to “add an Edge“
- Give it a name
- I would suggest, if you have this functionality, to tick the “HA” tickbox so that Edge is deployed in an “active/passive” fashion
- Provide credentials for the Edge device
- Select the uplink interface for this Edge
- Specify the default gateway
- Add the HA options, I would leave this set to the default
- And finish the config
Now if you had a virtual wire, and it needed to be connected to an Edge (more than likely) make sure to connect the virtual wire to the Edge by going back to “Networks”. Select the wire and then the “actions dial” and click “Connect to Edge” and select the correct edge device.
Now that you have a couple of wires you can start provisioning VMs or migrating VMs to them. Simply add them to the right network during the provisioning process.
Thanks Duncan!
Thanks for the writeup Duncan
One question though, what physical switches do you use in the lab and what config changes did you make on those switches? Things like MTU to 1600 and igmp-snooping and an igmp querier, or do you use a single VLAN to back all vWires and use jumboframes anyway?
In my scenario I used a single VLAN with the MTU configured to 1600 indeed. I don’t own the network and just request the network guys to make the changes required, which makes life a bit easier 🙂
Hi Duncan
In my environment, i cant change physical switch’s MTU Setting, can i just change the MTU to 1500 when preparing the environment ?
if i change vm’s MTU value to 1400, that would work ?
very appreciate for your reply
Any reason you didn’t refer to it as vCloud Networking and Security? I thought “vShield” was EOL – this may confuse some people.
I use the names as they are used in the product when you install / configure them. I know the marketing names are different, but as these are not reflected in the product yet I find it confusing to use them in the article.
Thanks for this nice write-up Duncan.
One other thing to be aware of is that during the preparation process, the “vib” package is downloaded from the vCenter server over port 80. I’ve ran into this when trying to configure VXLAN in a production environment where the ESXi hosts were deployed in a separate VLAN.
I was writing a new post with “requirements” in there, will add that.
It was a good article; however, you did not talk about configuring a firewall
for the vxlan’s based on some security group, etc.
Thanks
Chris
No I did not indeed. This is just about VXLAN wires, not about the possible edge services you can apply
Very useful article! Worked like a glitch.
Has anyone ever managed to get virtual guest tagging running with VXLAN? I want to use multiple VLANs tagged by a VM and put it into a VXLAN.
BR
/Peter
Great read Duncan.
Vxlan seems like the panacea and looks like a no-brainer to go with if you want to scale your data center yet maintain large mobility domains. The east west traffic understanding is simple enough. How does the north south traffic patterns works here when traffic comes into the data center destined for one of the vms on the vxwire?
Also, what would be the design and step through when vm sitting on one vxwrie wants to talk to a vm on another vxwire?
Hi Duncan, Any good input on how to change from etherchannel policy to “failover” when you have already created the VXLAN configuration?
Hi Duncan,
I am currently doing a Proof of Concept for vCloud, VXLAN is very good feature which remove the restricting of vlan’s and allows to scale-out. However for our current setup in production we do not need VXLAN, We are convinced that due to the critical data we handle we need the best network performance, so we definitely need vlan backed network however without VXLAN can this be possible?
FYI am using the VMware vCloud Direcor 5.1 Evaluation Guide for my setup and has skipped the VXLAN configuration from on page 40. On my test (POC) deployment I am getting an error:
{Ensure that infrastructure is prepared in VSM before creating or modifying VXLAN network pool. Cluster has not been prepared or mapped to a vDS in VSM.}
which is because I have not prepared my host for VXLAN.
I will appreciate if you can point me in the right direction to get the vCloud working without VXLAN.
Just select “VLAN” or “Portgroup” based…
Hi Duncan!
I’m working with Greg Hooven and Mary Vermokowitz to compile a contact list of all the bloggers we have on file, however we don’t currently have your email address. We will be sending a weekly email on news and updates as well as upcoming events related to the HP and VMware Alliance and want to make sure you are included. If you can send me your email address I’ll add you to the list.
Best Regards,
Michelle Hart
Hey Duncan,
Appreciate the help on the article! I am very new to vCloud Director and I was able to get everything deployed, but am having difficulties understanding how the VXLAN works when you want to add external access to other networks (subnets) to the mix (particiularlly, I want to give my VXLAN internet access).
I am currently using the default VXLAN that gets created when you provision a new organization inside of vCloud Director. Inside of my organization, I have created a new Org VDC Network with the DHCP address and Default Gateway, which gives me access to communicate internally between the VMs and the virtual gateway, however how do I now allow traffic to be forwarded from the virtual gateway to my physical firewall gateway to get out to the internet?
Thanks in advance!
Jack
Hi Duncan ,
I have created the VXLAN successfully and the VM is not getting any IP once it is assigned with the VXLAN Port group . What would be the next step to communicate between two VM on a cluster on same VXLAN network .
Regards
Hari Rajan