Yellow Bricks

by Duncan Epping

service console

VMReference Card 1.3

· ·

Forbes just released version 1.3 of his VMReference Card.This card can be very useful for those studying for VCP/VCDX. But it might also come in handy when doing troubleshooting on the Service Console:

I’ve just finished updating my reference card.  The biggest change is that I’ve moved everything to the latest update of 3.5 as the default.

  • Updated the details to the latest Confuguration Maximiums PDF.
  • Updated it to include 3.5 update 3 release notes.
  • Changed the versioning to include the latest VMware release, so its more obvious how up to date (or not) your card is.
  • Some minor additions (NAS maximums) and corrections.

Many thanks to all the readers who have written in with comments.  Always welcome.

Go and grab it here: http://www.vmreference.com/vi3-card

Go to VMReference.com and download the Card. Leave a comment while you are there!

Orphaned vmdk’s

· ·

While doing a “mini-healthcheck” at a customer site I noticed a specific Datastore with less than 2% of free diskspace. After a bit of research an orphaned VMDK was found. Orphaned vmdk’s are virtual hard-disks that are not connected to a VM. Probably because they were removed from the inventory without deleting the files.

You can easily find these orphaned vmdk’s via the Service Console:

find -iname “*-flat.vmdk” -mtime +7

For those that don’t like using the Service Console you can also check this with Powershell Ad van Bokhoven created a nice script which he describes as follows:

This script asks the virtual center what the disk are of each VM and puts this into an array. After this, it reads all files on all datastores. If the file is a vmdk file, it will check wheter this file is in the array. If it’s not, you’ve found a orphaned vmd.

I would advise to regularly check your environment on orphaned disks, it can save precious diskspace.

Tripwire Configcheck

· ·

When I published my article on tools/scripts I use during a VMware Healthcheck I received a couple of emails on Tripwire’s Configcheck. I’ve been on a holiday for a couple of weeks so it took me a bit longer than usual to check out the product.

Configcheck can be downloaded for free. Configcheck is a Java Application so you will need to install JRE. Installing JRE can be a bit of a pain sometimes on a server so this is one of the reasons for me that will make it hard to actively use Configcheck at customer sites. (This depends on the customers policy.) Installing the product is fairly easy though:

  1. Download Java JRE.
  2. Download the file configcheck.zip to a Windows machine that has Java Runtime Environment (JRE) version 1.5, or higher.
  3. Unzip the configcheck.zip file

That’s it, fairly easy. Now you can run “configcheck.cmd” to check the specified ESX host on security issues. Once the check is complete you can click the test results to view remediation steps. The test results will look like this:

As you can see, 37 Passed and 40 Failed. Not really surprising considering the fact that I ran this against a newly build ESX 3.5 U3 host. No modifications whatsoever. Clicking the test results didn’t work on my test servers because of the lack of an internet connection. Unfortunately it’s also not possible to export the data in this version. It’s free and Tripwire’s Enterprise edition does give you this capability, if you need export and a whole lot more check it out. You can find a data-sheet with a comparison between Configcheck and enterprise here.

Luckily Tripwire also provides the remediation steps in pdf form. For instance the remediation steps for 1.2.2 “Verify the log files to keep is equal to 10”:

Description: 
This test determines if virtual machines are configured to keep 10 log files when the recommended log rotate size of 100KB is exceeded. Virtual machines log activity in their respective vmware.log files. If growth of these log files is not limited, it is possible for virtual machines to cause a denial of service on the ESX Server by filling up the VMFS volume. There are two options for preventing virtual machines from flooding the hard disk of the host: size-based log file rotation or disabling logging for the virtual machine. This policy checks for size-based log file rotation because disabling logging altogether limits troubleshooting options.

Remediation:
To remediate failure of this policy test, configure the virtual machine to keep 10 log files when the recommended log rotate size of 100KB is exceeded. Configuring the virtual machine to keep 10 log files when the recommended log rotate size of 100KB is exceeded:

Login to the VirtualCenter or use the VI Client to connect directly to the ESX Server hosting the improperly configured virtual machine.

  1. Power off the virtual machine if needed.
  2. Right click the virtual machine and click Edit Settings.
  3. Select the Options tab.
  4. Select Advanced > General, and click the Configuration Parameters button.
  5. Look for a row with log.keepOld and set the value to 10.
  6. If the row does not exist, then click the Add Row button.
  7. In the Name field type log.keepOld.
  8. In the Value field type the value to 10.
  9. Click OK to close the Configuration Parameters dialog.
  10. Click OK to close the Virtual Machine Properties dialog.
As you can see, the description and remediation explain why and what to do in a fairly extensive manner. Which is great cause not does this make solving the “problem” really easy, Tripwire’s Configcheck also educates the SysAdmin!

Old School: Label your Hardware

· ·

So you were used to labelling your hardware with the name of the System running on it. But when running everything virtual you can label your ESX hosts but never know which VM resides at which Server without checking your console and/or vCenter.

Wouldn’t it be cool if you would have a magic Label that updated itself every once in a while. This way one would be able so see within just a glance which VM runs on which host.As you know there’s no such thing as a magic “label”, or maybe there is…

Yesterday I received an email from Nick Weaver(@lynxbat). He emailed me about a very very very cool script he wrote. No this script isn’t going to update your printed label off course. This script displays the VM’s running on your host on the front panel LCD. Most servers these days have frontpanel LCD’s and they can be updated with a couple simple ipmi commands.

Nick wrote an extensive article on how-to create a self updating magic label 🙂 in short:

  1. Install Dell OpenManage and run it on the ESX host (needed for ipmi drivers)
  2. Install ipmitool 1.8.10(SCP over, ./configure, make, make install….)
  3. Run lcd_update.sh script

Now walk over to your Dell server and check the result in the display, isn’t that amazing. Probably one of the most inventive scripts I’ve seen the last few months, it’s simple and gets the job done. Great job Nick, and I’m really curious what you will come up with next.

If I can find a Dell Machine this week I will definitely test it and post a screenshot!
I just received a link to a youtube video that shows it’s actually working!

vimsh, what can I do with it?

· ·

Vimsh(and vmware-vim-cm) is probably one of the worst documented commands out there. At the same time it’s one of the most powerful commands(I know it’s a shell…) out there. You name it and “vimsh” does it. Most of you ran into the “enabling vmotion” from the Service Console problem when first starting out with scripted install. Vimsh solves this:

/usr/bin/vmware-vim-cmd “hostsvc/vmotion/vnic_set vmk0″

As you can see “vimsh” is very powerful, but most of the other command-line stuff can be handled with the “esxcfg-*” commands. Well almost, for instance we talked about enabling autostart in my previous post. According to the KB article you must edit the file “/etc/vmware/hostd/vmAutoStart.xml”. Editing this file can be dangerous, I guess this goes for most ESX configuration files. During the Dutch VMUG I had a short chat with Wil van Antwerpen, Wil told me he was busy documenting the “vimsh” “command” in a wiki. After I published the enabling autostart blog Wil emailed me that this could and should be done with “vimsh”. I fully agree with Wil:

vmware-vim-cmd /hostsvc/autostartmanager/enable_autostart true

This enables the autostart functionality without manually editing the files. But I guess you would like to check if it’s enabled or disabled:

vmware-vim-cmd /hostsvc/autostartmanager/get_defaults

The big question remains, how do I know what I can and can’t do with “vimsh”. Well that’s the main reason for this post, as I said Wil has been very busy documenting “vimsh”. Wil created VI-Toolkit.com. VI-Toolkit.com contains a section on vimsh. Besides the the info that the vimsh command provices Wil added sample code. The sample code can be very usefull, but the search function is even more useful. Searching the vimsh documentation provides you with a fast way to check if a specific configuration action can be scripted with “vimsh”. For instance a search on “vimsh role” returns the following:

* Vimsvc/auth/role add
==== vimsh vimsvc/auth/role_add ==== Usage: role_add roleName [priv0] [priv1] [priv2] [priv3] [priv4]
171 B (24 words) – 14:32, 26 December 2008
* Vimsvc/auth/role permissions
==== vimsh vimsvc/auth/role_permissions ==== Usage: role_permissions roleName
1 KB (118 words) – 22:26, 28 December 2008
* Vimsvc/auth/role remove
==== vimsh vimsvc/auth/role_remove ==== Usage: role_remove roleName [failIfUsed]
123 B (16 words) – 14:34, 26 December 2008
* Vimsvc/auth/roles
==== vimsh vimsvc/auth/roles ==== Usage: roles
7 KB (550 words) – 21:50, 28 December 2008

I guess I can sum up this blog post in just one line:”Bookmark VI-Toolkit.com and add it to your RSS reader!”. Be sure to not miss out on anything regarding “vimsh” or any of the VI Toolkits that Wil be be describing and aggregating source code for. The “vimsh” section alone is already 345 pages large and it will continue to grow even more. Keep up the great work Wil and it was nice meeting you in person!