• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Yellow Bricks

by Duncan Epping

  • Home
  • Unexplored Territory Podcast
  • HA Deepdive
  • ESXTOP
  • Stickers/Shirts
  • Privacy Policy
  • About
  • Show Search
Hide Search

VMware

Multi NIC vMotion, how does it work?

Duncan Epping · Dec 14, 2011 ·

I had a question last week about multi NIC vMotion. The question was if multi NIC vMotion was a multi initiator / multi target solution. Meaning that, if available, on both the source and the destination multiple NICs are used for the vMotion / migration of a VM. Yes it is!

It is complex process as we need vMotion to able to handle mixes of 10GbE and 1GbE NICs.

When we start the process we will check, from the vCenter side, each host and determine the total combined pool of bandwidth available for vMotion. In other words, if you have 2x1GbE NICs and 1x10GbE NIC, then that host has a pool of 12GbE worth of bandwidth. We will do the same for the source and the destination host. Then, we will walk down each host’s list of vMotion vmknics, pairing off NICs until we’ve exhausted the bandwidth pool.

There are many combinations possible, but lets discuss a few just to provide a better idea of how this works:

  • If the source host has 1x1GbE NIC and the dest 1x1GbE NIC, we’ll open one connection between the these two hosts.
  • If the source has 3x1GbE NICs and the destination 1x10GbE NIC, then we’ll open one connection from each source-side 1GbE NIC to the destination’s 10GbE NIC – so a total of three socket connections all to the dest’s single 10GbE NIC.
  • If the source has 15x1GbE NICs and the destination 1x10GbE NIC and 5x1GbE NICs, then we’ll direct the first 10 source-side 1GbE NICs to connect to the dest’s 10GbE NIC, then the remaining pair of 5 1GbE vmknics will connect to each other – 15 connections in all.

Keep in mind that if the hosts are mismatched, we will create connections between vmknics until one of the sides is “depleted”. In other words if the source has 2 x 1GbE and the destination 1 x 1GbE only 1 connection would be opened.

 

Using Storage IO Control and Network IO Control together?

Duncan Epping · Dec 7, 2011 ·

I had a question today from someone who asked if there was any point in enabling SIOC (Storage IO Control) when you have NIOC (Network IO Control) enabled and configured. Lets start with the answer: Yes there is! NIOC controls traffic on a single NIC port level. In other words, when you have 10GbE NIC ports and vMotion, VMs and NFS (for instance) use the same NIC port it will prevent one of the streams from claiming all bandwidth while others need it. It basically is the police officer who controls a group of people getting too loud in a single room.

As not many people realize this lets repeat it… NIOC controls traffic on a NIC port level. Not on a NIC pair, not on a host level and not on a cluster wide level. On a NIC port level!

SIOC does IO control on a Datastore-VM layer. Meaning that when a certain threshold is reached it will determine on a datastore wide level which hosts and essentially which VMs get a specific chunk of the resources. SIOC prevents a single VM from claiming all IO resources for a datastore in a cluster. SIOC is cluster wide on a datastore level! It basically is the police officer who asks your neighbor to tone it down when as he is bothering the rest of the street.

Yes, enabling SIOC and NIOC together makes a lot of sense!

Some more nuggets about handling VIB files

Duncan Epping · Nov 30, 2011 ·

After I posted my article yesterday Jason Boche posted a comment about the reboot that was required according to the screenshot. I looked in to it and quickly realized that if I would alter my “descriptor.xml” I would not get this message. In other words, it depends on the package that is installed if a reboot is required or not, in my case I made the following simple changes to install the package without the need to reboot:

<live-install-allowed>true</live-install-allowed>
<live-remove-allowed>true</live-remove-allowed>

In other words, I am allowed to install it without a reboot and remove it without a reboot. Simple huh? Of course I tested it and this is the result:

~ # esxcli software vib install -v /test.vib
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: Duncan_bootbank_firewallrule_1.0
   VIBs Removed:
   VIBs Skipped:
~ #

After clicking refresh in the vCenter client the firewall rule I created pops up as expected.

Now if you would like to know before installing what the package contains and what the requirements are you can simply figure that out by doing the following:

~ # esxcli software sources vib get -v file:/test.vib
Duncan_bootbank_firewallrule_1.0
   Name: firewallrule
   Version: 1.0
   Type: bootbank
   Vendor: Duncan
   Acceptance Level: CommunitySupported
   Summary: Firewall rule
   Description: Firewall rule
   Release Date: 2011-06-01
   Depends:
   Conflicts:
   Replaces:
   Provides:
   Maintenance Mode Required: False
   Hardware Platforms Required:
   Live Install Allowed: True
   Live Remove Allowed: True
   Stateless Ready: False
   Overlay: False
   Tags: driver, module
   Payloads: test
~ #

As you can see in this case, “live install allowed” is set to true. The vendor is “Duncan” and the Acceptance Level is “CommunitySupported”, these are important details in my opinion! Another one to keep an eye on is if the package is “Stateless Ready” or not. In my case I defined it as “false”.

Of course you can also remove a VIB file after installing it. This is pretty straight forward, first of all list all the installed VIBs:

~ # esxcli software vib list
Name                  Version                             Vendor  Acceptance Level  Install Date
--------------------  ----------------------------------  ------  ----------------  ------------
ata-pata-amd          0.3.10-3vmw.500.0.0.456551          VMware  VMwareCertified   2011-06-06
ata-pata-atiixp       0.4.6-3vmw.500.0.0.456551           VMware  VMwareCertified   2011-06-06
ata-pata-cmd64x       0.2.5-3vmw.500.0.0.456551           VMware  VMwareCertified   2011-06-06

After listing all installed VIBs you can easily remove them by using the following command:

~ # esxcli software vib remove -n ata-pata-amd

This would remove the VIB named “ata-pata-amd”. You could even do a “dry-run” to see what the result would be:

~ # esxcli software vib remove -n ata-pata-amd --dry-run
Removal Result
   Message: Dryrun only, host not changed. The following installers 
   will be applied: [BootBankInstaller]
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.500.0.0.456551
   VIBs Skipped:
~ #

I hope this provides some more details around how to handle VIB files. Don’t hesitate to leave a comment if you have any questions at all.

How to create your own .vib files

Duncan Epping · Nov 29, 2011 ·

** Be warned, this is totally unsupported. Only for educational purposes should this be used **

Today I was asked the question on how to create a VIB file (.vib). In our documentation it is mentioned that you can create a VIB file to add firewall rules to your ESXi host. As the .vib tool is not available yet to the general public I decided to dig in to it. I want to stress that I tested this in my own lab, it is not supported at all, but might give a nice insight in how these VIB are constructed. Before you read how I created my own VIB file I suggest reading this excellent article on what a .vib file is and contains by my colleague Kyle Gleed.

First thing I did was download an existing VIB file. I downloaded a tiny LSI SCSI driver. I did a “more” of the .vib file and I noticed the following:

!<arch>
debian-binary

That was my first lead, it appears to be a debian-binary, which is a format that the Linux distribution Debian uses to package software / drivers etc. I knew it should be possible to check what was included in this package. So I did a quick search and stumbled on some procedures on how to do this using some standard commands provided by my Debian virtual machine. (Links at the bottom) So I did the following on the package I downloaded:

ar tv file.vib

This showed me that the .vib file contained three files:

descriptor.xml
sig.pkcs7
scsi-meg

This seemed pretty obvious to me after reading Kyle’s article. The descriptor contained the metadata, the “sig*” file contained the signature and the “scsi-meg” was the actual driver. I decided to extract the VIB file to look at the content of these files:

ar vx file.vib

As the permissions on the files didn’t allow me to look at them I changed the permissions on those by using “chmod”. Now what? Well let’s look at the “scsci-meg” file first. What is it? I looked at what was in the file by using the following command:

tar -tzvf scsi-meg

It contained a list of files and that is it. I decided to extract it using “tar -xzvf” and as expected it was the folder structure and files part of this driver. I figured that it wouldn’t be too difficult to create a simple package. Why not try it… Here we go. First I deleted everything in the “sig.pkcs7” file. As Kyle mentioned in his article that community support packages can have an empty signature. I also deleted all the files and folders that were extracted from the “scsi-meg” package that I did not need. I then created a folder underneath the “/etc/vmware” structure as I wanted to create a firewall rule. (Added the folder “firewall”.)

I copied a firewall rule from my existing ESXi host and which is created by HA to my Debian VM and edited the file, the original file was “fdm.xml”. I edited and and renamed it to test.xml. I changed all ports to 7000 and changed the <id> of the service that would need to be added and saved the file in “etc/vmware/firewall”.

Now it was time to package it all up and see if it would work. I guessed that the steps required would simply be the reverse of what I did to extract it all.

tar -czvf etc/ test

I then opened up the descriptor.xml file and changed some of the fields around, most don’t seem to matter much except for the following:

Change the following key to:
<acceptance-level>certified</acceptance-level>
<acceptance-level>community</acceptance-level>
Add your list of files:
<file-list>
<file>path-to-file</file>
</file-list>
Change the name of your package and the size accordingly:
<payload name="test" type="vgz" size="809">

I wasn’t sure if that would work, but I would find out eventually I guess (yes I also tried “communitysupport” as the acceptance-level but that doesn’t work!). I also removed the checksum details from the descriptor file just in case it would be used. This is what my full descriptor file looked like:

<vib version="5.0">
<type>bootbank</type>
<name>firewallrule</name>
<version>1.0</version>
<vendor>Duncan</vendor>
<summary>Firewall rule</summary>
<description>Firewall rule</description>
<release-date>2011-06-01T22:16:31.062257+00:00</release-date>
<urls/>

<relationships>
<depends>
</depends>
<conflicts/>
<replaces/>
<provides/>
<compatibleWith/>
</relationships>

<software-tags>
<tag>driver</tag>
<tag>module</tag>
</software-tags>

<system-requires>
<maintenance-mode>true</maintenance-mode>
</system-requires>

<file-list>
<file>etc/vmware/firewall/test.xml</file>
</file-list>

<acceptance-level>community</acceptance-level>
<live-install-allowed>false</live-install-allowed>
<live-remove-allowed>false</live-remove-allowed>
<cimom-restart>false</cimom-restart>
<stateless-ready>false</stateless-ready>
<overlay>false</overlay>

<payloads>
<payload name="test" type="vgz" size="809">
</payload>
</payloads>
</vib>

Next up would be making a single .vib file out of these three components again:

ar -r test.vib test descriptor.xml sig.pkcs7

Now I need to ‘scp’ the file to my ESXi host and see if I can install it:

scp test.vib root@esxi:test.vib
esxcli software vib install -v /test.vib

I received an error that the ImageProfile acceptance level needed to be changed. That was my next step:

esxcli software acceptance set --level CommunitySupported

After repeating the “esxcli software vib install” command I received the following output:

~ # esxcli software vib install -v /test.vib
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: Duncan_bootbank_firewallrule_1.0
   VIBs Removed:
   VIBs Skipped:
~ #

I rebooted the host and here’s a screenshot of the ESXi firewall with the newly added custom service “Test”:

Once again, I want to point out that this is currently unsupported. Don’t use this in your production environment!

The following articles helped me figuring this out and producing this article:

http://tldp.org/HOWTO/html_single/Debian-Binary-Package-Building-HOWTO/

http://linuxtrove.com/wp/?p=78

Crazy Black Friday / Cyber Monday deal!

Duncan Epping · Nov 25, 2011 ·

I know there are a lot of crazy deals on Black Friday and Cyber Monday, but this is most definitely the deal of the year for all virtualization fanatics! I just changed the price of the vSphere 5 Clustering Technical Deepdive to $ 4.99 and 3.99 for my European friends. Yes that is correct…. Less than 5 dollars for over 350 pages of deepdive material. Keep in mind that this is a limited offer, Tuesday the 29th the price will be back to “normal” again.

US – ebook – $ 4.99

UK – ebook – £ 3.99

DE – ebook – € 3.99

FR – ebook – € 3.99

Pick it up, tell your friends / colleagues / family about it… Here are some snippets from Amazon reviews, but with 15 extremely positive reviews, all of them 5 out of 5, you know you can’t go wrong:

“If you’re serious about VMware virtualization this book is a must have. Regardless of you responsibilities with a virtual infrastructure administrative, or from a architecture design stand point this book is for you. The level of knowledge and depth which Frank and Duncan cover in this book about the new clustering changes in vSphere 5 is priceless. The design tips and illustrations through the book are truly invaluable. There is no other book that gets into the core of all the different vSphere 5 cluster technologies like this one, ”

“Whether you are longing to know about the transition from AAM to FDM, best practices for DRS and DPM, or are just curious to know what those acronyms are this is a great book! The technical detail, practical advice, and comparative analysis throughout make this book one of the most thorough yet concise technical books available.”

“The book is clearly written, a special emphasis has been made on making it understandable even for professionals like me who use vSphere daily yet do not manage huge production environments. The book goes to great lengths to explain all possible scenarios and I found answers to all my questions. Not only sections cover HOW the technology works, but the authors go as far as explaining the way the algorithms are working, which will satisfy the curiosity of everyone.”

“The complete explanations provide the reader all of the information needed to make informed decisions about their environment with excellent diagrams to provide strong visual reinforcements.”

** Please note that Amazon charges for remote downloads, so the price listed here is “local” download price without the additional Amazon transfer costs **

 

  • « Go to Previous Page
  • Page 1
  • Interim pages omitted …
  • Page 79
  • Page 80
  • Page 81
  • Page 82
  • Page 83
  • Interim pages omitted …
  • Page 123
  • Go to Next Page »

Primary Sidebar

About the Author

Duncan Epping is a Chief Technologist and Distinguished Engineering Architect at Broadcom. Besides writing on Yellow-Bricks, Duncan is the co-author of the vSAN Deep Dive and the vSphere Clustering Deep Dive book series. Duncan is also the host of the Unexplored Territory Podcast.

Follow Us

  • X
  • Spotify
  • RSS Feed
  • LinkedIn

Recommended Book(s)

Advertisements




Copyright Yellow-Bricks.com © 2025 · Log in