As far as I know there’s another way to show this:

cat /etc/vmware/esx.conf | grep "teamPolicy\/team"

This would return the following:

/net/vswitch/child0001/teamPolicy/team = "lb_srcid"

Possible load balancing policies:
lb_srcid = Virtual port id
lb_ip = IP Hash
lb_srcmac = MAC Address

If anyone knows of any other methods of showing this info let me know and/or contribute to the topic on VMTN.

Update:

@lamw just posted that it is possible to get this info from vmware-vim-cmd:

vmware-vim-cmd hostsvc/net/vswitch_info | grep -E '(policy|name)'
      name = "vSwitch0",
         policy = (vim.host.NetworkPolicy) {
               policy = "loadbalance_ip",
      name = "vSwitch1",
         policy = (vim.host.NetworkPolicy) {
               policy = "loadbalance_srcid",
      name = "vSwitch5",
         policy = (vim.host.NetworkPolicy) {
               policy = "loadbalance_srcid",

1. vmware-vim-cmd vimsvc/license --source server

2. vmware-vim-cmd vimsvc/license --source file

Don’t you just love “vmware-vim-cmd”….

When a VM gets stuck during shutdown or isn’t responding anymore you can easily kill the VM. First option is the command line version of vCenter’s “shutdown vm”, first list all the VMs running on the host so you can copy and paste the <config> in to the next command. The command “vmware-cmd <config> stop trysoft” will try to initiate a soft shutdown first, in other words a shutdown via the Guest OS, if that doesn’t work it will do a power off. Now, as most of you probably already experienced, sometimes it’s impossible to shutdown the VMs in a normal way. This is where 2nd, 3rd and 4th option come in to place. Option two uses vm-support to kill the VM, use “-x” to list the VM id’s and kill it with “-X”. The third option uses vimsh, in this case we use vmware-vim-cmd, “vmsvc/getallvms” lists all VMs and the id’s and with “vmsvc/poweroff” you can specify the VM that needs to be powered off. The fourth option is the Linux/Unix way of doing it, find the process id of the VM via “ps -auxwww” and just kill it.

1. vmware-cmd -l

   vmware-cmd <config> stop trysoft

2. vm-support -x

   vm-support -X <vmid>

3. vmware-vim-cmd vmsvc/getallvms

   vmware-vim-cmd vmsvc/poweroff <vm id>

4. ps -auxwww | grep <vm name>

   kill <process id>

5. if option three isn’t successful do the following:

   kill -9 <process id>

As VMwareWolf points out, there’s an excellent KB article on this subject to be found here: http://kb.vmware.com/kb/1004340

Both configurations are supported by VMware and each has their own pros and cons:

1. Requirements: 2 physical nics, vlan’s and vlan trunking:
   • vSwitch0: 2 Physical nics(vmnic0 & vmnic2)
   • 2 Portgroups(Service Console & VMkernel)
   • Service Console active on vmnic0 and standby on vmnic2
   • VMkernel active on vmnic2 and standby on vmnic0

   Each portgroup has a vlan ID assigned and runs dedicated on its own physical NIC, only in the case of a fault it’s switched over to the standby NIC. It will return to the original NIC when the physical NIC is up and running again. This is achieved by setting “Failback” to Yes!

   Pros: Only need 2 NICs in total for the Service Console and VMkernel, especially useful in Blade environments. Simple setup.

   Cons: If the connection is dropped several times it will cause the NIC to fail-over each time which can cause HA to kick in eventually. You will need to set the Failure Detection Time (das.failuredetectiontime) to 60 seconds opposed to the default of 15 seconds. (According to VMware best practices.)

2. Requirements: 3 physical NICs, vlan’s and vlan trunking:
   • vSwitch0 – 1 Physical NIC(vmnic0) – 1 Portgroup(Service Console)
   • vSwitch1 – 2 Physical NICs(vmnic1 & vmnic3)
   • 2 Portgroups (VMkernel & Secondary Service Console)

   The primary Service Console runs dedicated on a physical NIC, vmnic0, with a vlan assigned on either the physical switch port or the portgroup. (I would prefer portgroup for consistency).

   The second vSwitch, vSwitch1, will run the VMkernel active on vmnic1 and standby on vmnic3. The secondary Service Console will be active on vmnic3. For the secondary Service Console I would prefer to set the vmnic1 to “unused”, this way you are 100% sure that the Service Console will run only on vmnic0 and vmnic3.

   Pros: a lower “failure detection time”(das.failuredetectiontime) can be defined this is because of the fact that the service console is already active. Here the Failure Detection Time can be set to 20 seconds, further no Spanning Tree problems will occur as the setup contains two service consoles, and subsequently 2 MAC addresses. (This is also the reason why I would set vmnic1 to unused for this portgroup)

   Cons: Need to set extra isolation addresses(das.isolationaddress), and secondary Service Console preferably in a different subnet.

Thanks goes out to mr Tom Howarth for a sanity/grammar check.

Yesterday I encountered a performance issue at a customer site. One thing I’ve learned over the last couple of years is that “ESXTOP” can be very useful in pinpointing performance issues, so writing this article happened sooner than I expected. The customer measured all sorts of counters within the VM and all the symptoms made the customer conclude that the problem was related to the virtual SCSI controller and / or the virtual harddisks(vmdk’s). The symptoms were high “Physical Disk\Avg. Disk sec/Transfer” and peak “Physical Disk\Avg. Disk Writes/Sec” behaviour. In other words, transferring data to and from the disk took too long and there wasn’t a constant stream of I/O.

The initial conclusion that it was VMDK / SCSI Controller related isn’t weird at all. Looking at the values of the VM itself I would also suspect that this was the case. But ESXTOP revealed something totally different. There wasn’t a high “disk queue” or heavy I/O for this particular VM or host.

The “C” shows all the CPU information with a breakdown in worlds and groups. For more info on this check this Community Document. CPU revealed a high %RDY value. %RDY is also explained in the community document, besides all the other relevant values. In short what %RDY represents:

A world in a run queue is waiting for CPU scheduler to let it run on a PCPU. %RDY accounts the percentage of this time. So, it is always smaller than 100%.

Best practice would be values of around 5%, an occasional max of 10-20% wouldn’t hurt most VM’s / Applications. In this case the value was at least 50% all of the time.

Next question of course was “what causes this high ready times”. This one was easy to answer/explain: overprovisioning. The host had 8 cores (2 x quad core) and currently running on the host: 8 VM’s with a total of 18 vCPU’s. In other words, most VM’s were provisioned with multiple vCPU’s which makes scheduling even more difficult. Especially when you’ve got multiple 4 vCPU VM’s running. And when scheduling is successful the task would still have to wait for the complete socket to be available when there are no “idle loops” running within the VM vCPU’s.

One socket indeed, that’s based on the scheduler cell size which is 4 by default. In other words:
2 x dual core = 1 cell
1 x quad core = 1 cell
2 x quad core = 2 cell’s (each proc = 1 cell, vm’s can’t span multiple cell’s)

Each vCPU will be tied to a core, meaning that with 4 vCPU’s one would need 4 Cores, 1 complete cell available. (For those thinking I’ve got 6-core processors, there’s a way to increase the scheduler cell size! In short set vmkernel.boot.cpucellsize to 6) Conclusion of this very usefull day, be very careful when provisioning your VM’s. Do they really need 4 vCPU’s, or even 2 vCPU’s… and when you do provision them with multiple vCPU’s monitor these VM’s. Claim the vcpu’s back when these aren’t used or you don’t get the expected performance results.

I’ve just finished updating my reference card.  The biggest change is that I’ve moved everything to the latest update of 3.5 as the default.

• Updated the details to the latest Confuguration Maximiums PDF.
• Updated it to include 3.5 update 3 release notes.
• Changed the versioning to include the latest VMware release, so its more obvious how up to date (or not) your card is.
• Some minor additions (NAS maximums) and corrections.

Many thanks to all the readers who have written in with comments.  Always welcome.

Go and grab it here: http://www.vmreference.com/vi3-card

Go to VMReference.com and download the Card. Leave a comment while you are there!

You can easily find these orphaned vmdk’s via the Service Console:

find -iname “*-flat.vmdk” -mtime +7

For those that don’t like using the Service Console you can also check this with Powershell Ad van Bokhoven created a nice script which he describes as follows:

This script asks the virtual center what the disk are of each VM and puts this into an array. After this, it reads all files on all datastores. If the file is a vmdk file, it will check wheter this file is in the array. If it’s not, you’ve found a orphaned vmd.

I would advise to regularly check your environment on orphaned disks, it can save precious diskspace.

Configcheck can be downloaded for free. Configcheck is a Java Application so you will need to install JRE. Installing JRE can be a bit of a pain sometimes on a server so this is one of the reasons for me that will make it hard to actively use Configcheck at customer sites. (This depends on the customers policy.) Installing the product is fairly easy though:

1. Download Java JRE.
2. Download the file configcheck.zip to a Windows machine that has Java Runtime Environment (JRE) version 1.5, or higher.
3. Unzip the configcheck.zip file

That’s it, fairly easy. Now you can run “configcheck.cmd” to check the specified ESX host on security issues. Once the check is complete you can click the test results to view remediation steps. The test results will look like this:

As you can see, 37 Passed and 40 Failed. Not really surprising considering the fact that I ran this against a newly build ESX 3.5 U3 host. No modifications whatsoever. Clicking the test results didn’t work on my test servers because of the lack of an internet connection. Unfortunately it’s also not possible to export the data in this version. It’s free and Tripwire’s Enterprise edition does give you this capability, if you need export and a whole lot more check it out. You can find a data-sheet with a comparison between Configcheck and enterprise here.

Luckily Tripwire also provides the remediation steps in pdf form. For instance the remediation steps for 1.2.2 “Verify the log files to keep is equal to 10″:

Description: 
This test determines if virtual machines are configured to keep 10 log files when the recommended log rotate size of 100KB is exceeded. Virtual machines log activity in their respective vmware.log files. If growth of these log files is not limited, it is possible for virtual machines to cause a denial of service on the ESX Server by filling up the VMFS volume. There are two options for preventing virtual machines from flooding the hard disk of the host: size-based log file rotation or disabling logging for the virtual machine. This policy checks for size-based log file rotation because disabling logging altogether limits troubleshooting options.

Remediation:
To remediate failure of this policy test, configure the virtual machine to keep 10 log files when the recommended log rotate size of 100KB is exceeded. Configuring the virtual machine to keep 10 log files when the recommended log rotate size of 100KB is exceeded:

Login to the VirtualCenter or use the VI Client to connect directly to the ESX Server hosting the improperly configured virtual machine.

1. Power off the virtual machine if needed.
2. Right click the virtual machine and click Edit Settings.
3. Select the Options tab.
4. Select Advanced > General, and click the Configuration Parameters button.
5. Look for a row with log.keepOld and set the value to 10.
6. If the row does not exist, then click the Add Row button.
7. In the Name field type log.keepOld.
8. In the Value field type the value to 10.
9. Click OK to close the Configuration Parameters dialog.
10. Click OK to close the Virtual Machine Properties dialog.

Wouldn’t it be cool if you would have a magic Label that updated itself every once in a while. This way one would be able so see within just a glance which VM runs on which host.As you know there’s no such thing as a magic “label”, or maybe there is…

Yesterday I received an email from Nick Weaver(@lynxbat). He emailed me about a very very very cool script he wrote. No this script isn’t going to update your printed label off course. This script displays the VM’s running on your host on the front panel LCD. Most servers these days have frontpanel LCD’s and they can be updated with a couple simple ipmi commands.

Nick wrote an extensive article on how-to create a self updating magic label in short:

1. Install Dell OpenManage and run it on the ESX host (needed for ipmi drivers)
2. Install ipmitool 1.8.10(SCP over, ./configure, make, make install….)
3. Run lcd_update.sh script

Now walk over to your Dell server and check the result in the display, isn’t that amazing. Probably one of the most inventive scripts I’ve seen the last few months, it’s simple and gets the job done. Great job Nick, and I’m really curious what you will come up with next.

If I can find a Dell Machine this week I will definitely test it and post a screenshot!
I just received a link to a youtube video that shows it’s actually working!

/usr/bin/vmware-vim-cmd “hostsvc/vmotion/vnic_set vmk0″

As you can see “vimsh” is very powerful, but most of the other command-line stuff can be handled with the “esxcfg-*” commands. Well almost, for instance we talked about enabling autostart in my previous post. According to the KB article you must edit the file “/etc/vmware/hostd/vmAutoStart.xml”. Editing this file can be dangerous, I guess this goes for most ESX configuration files. During the Dutch VMUG I had a short chat with Wil van Antwerpen, Wil told me he was busy documenting the “vimsh” “command” in a wiki. After I published the enabling autostart blog Wil emailed me that this could and should be done with “vimsh”. I fully agree with Wil:

vmware-vim-cmd /hostsvc/autostartmanager/enable_autostart true

This enables the autostart functionality without manually editing the files. But I guess you would like to check if it’s enabled or disabled:

vmware-vim-cmd /hostsvc/autostartmanager/get_defaults

The big question remains, how do I know what I can and can’t do with “vimsh”. Well that’s the main reason for this post, as I said Wil has been very busy documenting “vimsh”. Wil created VI-Toolkit.com. VI-Toolkit.com contains a section on vimsh. Besides the the info that the vimsh command provices Wil added sample code. The sample code can be very usefull, but the search function is even more useful. Searching the vimsh documentation provides you with a fast way to check if a specific configuration action can be scripted with “vimsh”. For instance a search on “vimsh role” returns the following:

* Vimsvc/auth/role add
==== vimsh vimsvc/auth/role_add ==== Usage: role_add roleName [priv0] [priv1] [priv2] [priv3] [priv4]
171 B (24 words) – 14:32, 26 December 2008
* Vimsvc/auth/role permissions
==== vimsh vimsvc/auth/role_permissions ==== Usage: role_permissions roleName
1 KB (118 words) – 22:26, 28 December 2008
* Vimsvc/auth/role remove
==== vimsh vimsvc/auth/role_remove ==== Usage: role_remove roleName [failIfUsed]
123 B (16 words) – 14:34, 26 December 2008
* Vimsvc/auth/roles
==== vimsh vimsvc/auth/roles ==== Usage: roles
7 KB (550 words) – 21:50, 28 December 2008

I guess I can sum up this blog post in just one line:”Bookmark VI-Toolkit.com and add it to your RSS reader!”. Be sure to not miss out on anything regarding “vimsh” or any of the VI Toolkits that Wil be be describing and aggregating source code for. The “vimsh” section alone is already 345 pages large and it will continue to grow even more. Keep up the great work Wil and it was nice meeting you in person!