Yellow Bricks

by Duncan Epping

Server

Installing drivers during a scripted install of ESXi

· ·

As you hopefully have read I have been busy over the last weeks with a new project. This project is all about enabling migrations to ESXi. I wrote two articles for the ESXi Chronicles blog of which the first article describes the scripted install procedure and the seconds gives some additional advanced examples of what is possible in these scripts. Based on that article I started receiving some questions from the field and last week I had a conference call with a customer who had issues “injecting” a driver into ESXi during the install. Normally installing a driver is done by using a simple “esxupdate” command and a reboot, but in this case however the situation was slightly different and let me explain the problem first.

This customer implemented a script that would run during the %firstboot section. The name of the section already explains what it is, this section will run after the reboot of the scripted install has been completed. The way this works is that ESXi creates a script in /etc/vmware/init/init.d with the prefix 999. This script will run as the last script during the boot and is generally used to configure the host. After a final reboot this script is automatically deleted and the host is ready to be added to vCenter.

The challenge however that this customer was facing is that it was using a Xsigo network environment. In this scenario a server that would need to be reinstalled would get a temporary network configuration that would only work during the first boot. Meaning that before this %firstboot section would even run the original network configuration would be restored. The problem with that however is that the original network configuration requires the drivers to be installed before it can be used. In other words, after the reboot done by the installer you will not be able to access the network unless you have loaded the drivers. This rules out downloading the drivers during the %firstboot section. Now how do we solve this?

The scripted installation has multiple sections that can contain your commands. The first section is called %post. The ESXi setup guide describes this section as follows:

Executes the specified script after package installation has been completed. If you specify multiple %post sections, they are executed in the order they appear in the installation script.

This means that in the case of this customer we will be able to use this section to download any driver package required with for instance “wget” and that is what I did. I issued the command below during the %post section and rebooted the server.

wget http://192.168.1.100/xsigo.zip

The problem however was that the package wasn’t persisted after a reboot which brought me back to right where I began, without a network after the restart. Than I figured that during the install a local datastore is created and I could use that as persistent storage. So I issued the following command in the %post section of the script:

wget http://192.168.1.100/xsigo.zip -O /vmfs/volumes/datastore1/xsigo.zip

I rebooted the installer and checked after the install if the driver bundle was there or not, and yes it was. The only thing left to do was to install the driver in the %fistboot section. This by itself is a fairly simple task:

esxupdate --bundle=/vmfs/volumes/datastore1/xsigo.zip update

As the host will need to reboot before the drivers are loaded I also added a “reboot” command at the end of to the “%firstboot” section. This ensures the drivers are loaded and the network is accessible.

I guess this demonstrates how easy a solution can be. When I first started looking into this issue and started brainstorming I was making things way too complex. By using the standard capabilities of the scripted install mechanism and a simple “wget” command you can do almost everything you need to do during the install. This also removes the need to fiddle around with injecting drivers straight into the ISO itself.

Go ESXi 🙂

Blocksize impact?

· ·

I was answering a couple of questions on the VMTN forum and stumbled on the impact of using similar blocksizes for all VMFS volume which I never realized. Someone did a zero out on their VM to reclaim some wasted diskspace by doing a Storage vMotion to a different datastore. (Something that I have described in the past here, and also with relationship to VCB here.) However to the surprise of this customer the zeroed out space was not reclaimed even though he did a Storage vMotion to a thin disk.

After some tests they noticed that when doing a Storage vMotion between Arrays or between datastores formatted with a different blocksize they were able to reclaim the zeroed out diskspace. This is a cool workaround to gobble the zeroes Now the question of course remains why this is the case. The answer is simple: there is a different datamover used. Now this might not say much to some so let me explain the three types that can be used:

  • fsdm – This is the legacy datamover which is the most basic version and the slowest as the data moves all the way up the stack and down again.
  • fs3dm – This datamover was introduced with vSphere 4.0 and contained some substantial optimizations where data does not travel through all stacks.
  • fs3dm – hardware offload – This is the VAAI hardware offload full copy that is leveraged and was introduced with vSphere 4.1. Maximum performance and minimal host CPU/Mem overhead.

I guess it is a bit easier to digest when it is visualized. It should be noted here that fs3dm is used in software mode when HW Offload (VAAI) capabilities are not available. Also note that “application”, the top layer, can be Storage vMotion or for instance vmkfstools:


(Please note that the original diagram came from a VMworld preso (TA3220))

Now why does it make a difference? Well remember this little line in the VAAI FAQ?

  • The source and destination VMFS volumes have different block sizes

As soon as different blocksized vmfs volume or a different array is selected as the destination the hypervisor uses the legacy datamover (fsdm). The advantage of this datamover, and it is probably one of the few advantages, is that it gobbles the zeroes. The new datamover (fs3dm), and that includes both the software datamover and the HW Offload, will not do this currently and as such all blocks are copied. The fs3dm datamover however is substantially faster than the fsdm datamover though. The reason for this is that the fsdm datamover reads the block in to the applications buffer (Storage vMotion in our case) and then write the block. The new datamover just moves the block without adding it to the application buffer. I guess it is a “minor” detail as you will not do this on a day to day basis, but when you are part of operations it is definitely a “nice to know”.

vSphere 4.1 Performance Troubleshooting Guide

· ·

Today I received an email from Chethan Kumar who is part of Performance Engineering. Chethan rewrote the vSphere Performance Troubleshooting guide so that it contains all the nitty gritty details there are to know about vSphere 4.1 when it comes to performance related problems.

source
The hugely popular Performance Troubleshooting for VMware vSphere 4 guide is now updated for vSphere 4.1 . This document provides step-by-step approach for troubleshooting most common performance problems in vSphere-based virtual environments. The steps discussed in the document use performance data and charts readily available in the vSphere Client and esxtop to aid the troubleshooting flows. Each performance troubleshooting flow has two parts:

  1. How to identify the problem using specific performance counters.
  2. Possible causes of the problem and solutions to solve it.

Now the cool thing about this document is that it not only shows you some of the key metrics to look at but it also provides you with workflows that contains the steps required to analyze these types of issues and of course ultimately solve them. Just download the paper and as an example look at figure 3 on page 16 or figure 5 on page 19 and you’ll know why I love this document.

This performance troubleshooting guide is one of the most invaluable documents out there as it will also give you a better in-sight of how some of the metrics are constructed. Something that I did not know but just discovered is the following:
If you have a snapshot and are experiencing high I/O response times validate the following:

LAT/rd and LAT/wr columns indicate the average response time of read and write I/O commands seen by the VM. Compare these with DAVG/rd and DAVG/rw. If LAT/rd > DAVG/rd or LAT/rw > DAVG/rw and QUED = 0 than the latency is more than likely cause by the snapshot as DAVG/rd and DAVG/rw is device latency and LAT/rd and LAT/rw is the latency observed by the guest.

You see, you can learn something new every day.

Anti-virus and the impact in virtualized environments

· ·

I was reading Richard Garsthagen’s article about anti-virus solutions yesterday and decided that this deserved a little bit of extra attention as it is an often overlooked area when it comes to architecture and impact. As Richard points out the difference in terms of load that it generates and overhead is enormous. All of these combined will most definitely result in an increase of consolidation ratio. Not only that but is will also seriously lower the risk during for instance a VDI boot storm but also think about the impact of HA initiated restarts. This could cause an enormous amount of IOps and CPU/Memory overhead which in its turn could impact the other virtual machines.

I guess there is no point in rehashing what is written in the whitepaper of what Richard wrote, I just want to point out the whitepaper as I believe it is a good read. As always results may vary but it is pretty obvious that from an architectural and operational perspective End Point Security is most definitely worth looking into and I cannot wait for more vendors to jump on the bandwagon. Download the tolly report here. (I personally found the disk results very interesting…)

E-Book version of the HA/DRS Deepdive?

· ·

I’ve had this question too many times over the last couple of weeks and instead of answering all of them separately I figured I would dedicate a blog article to it so that I can reference to that. Here we go:

Q) Will the e-book version of the HA/DRS deepdive be available soon?

A) No. We have decided that we will not release an e-book (including pdf, ibook, kindle) version of the book. It required a full reformatting of the book which is simple not feasible at this point in time. As it is a “self published” book we are responsible for not only the content but also the formatting etc. Sorry about that, for “Volume 2” however we will reconsider this.

 

I hope this answers your question(s). Thanks and again our apologies, the only option is the paper version.

UPDATE: http://www.yellow-bricks.com/2011/04/05/what-an-ebook-is-this-a-late-april-fools-joke/