With ESXi 5.0 this has been simplified, this is what it looks like today:

I also want to point out that many of the standard installation commands have been replaced, removed or are not supported anymore. I created a simple script to automatically install an ESXi 5.0 host. It creates a second vSwitch and a second VMkernel for vMotion. It enables both the local and remote TSM and sets the default PSP for the EMC VMAX to Round Robin. As you can see there is a huge shift in this script towards esxcli. Although some of the old “esxcfg-*” commands might still be working they are deprecated and no longer supported. The new standard is esxcli, make sure you get familiarized with it and start using it today as over time this will be the only CLI tool available.


# Sample scripted installation file
# Accept the VMware End User License Agreement
vmaccepteula
# Set the root password for the DCUI and ESXi Shell
rootpw mypassword
# Install on the first local disk available on machine
install --firstdisk --overwritevmfs
# Set the network to DHCP on the first network adapater, use the specified hostname and do not create a portgroup for the VMs
network --bootproto=dhcp --device=vmnic0 --addvmportgroup=0
# reboots the host after the scripted installation is completed
reboot

%firstboot --interpreter=busybox
# Add an extra nic to vSwitch0 (vmnic2)
esxcli network vswitch standard uplink add --uplink-name=vmnic2 --vswitch-name=vSwitch0
#Assign an IP-Address to the first VMkernel, this will be used for management
esxcli network ip interface ipv4 set --interface-name=vmk0 --ipv4=192.168.1.41 --netmask=255.255.255.0 --type=static
# Add vMotion Portgroup to vSwitch0, assign it VLAN ID 5 and create a VMkernel interface
esxcli network vswitch standard portgroup add --portgroup-name=vMotion --vswitch-name=vSwitch0
esxcli network vswitch standard portgroup set --portgroup-name=vMotion --vlan-id=5
esxcli network ip interface add --interface-name=vmk1 --portgroup-name=vMotion
esxcli network ip interface ipv4 set --interface-name=vmk1 --ipv4=192.168.2.41 --netmask=255.255.255.0 --type=static
# Enable vMotion on the newly created VMkernel vmk1
vim-cmd hostsvc/vmotion/vnic_set vmk1
# Add new vSwitch for VM traffic, assign uplinks, create a portgroup and assign a VLAN ID
esxcli network vswitch standard add --vswitch-name=vSwitch1
esxcli network vswitch standard uplink add --uplink-name=vmnic1 --vswitch-name=vSwitch1
esxcli network vswitch standard uplink add --uplink-name=vmnic3 --vswitch-name=vSwitch1
esxcli network vswitch standard portgroup add --portgroup-name=Production --vswitch-name=vSwitch1
esxcli network vswitch standard portgroup set --portgroup-name=Production --vlan-id=10
# Set DNS and hostname
esxcli system hostname set --fqdn=esxi5.localdomain
esxcli network ip dns search add --domain=localdomain
esxcli network ip dns server add --server=192.168.1.11
esxcli network ip dns server add --server=192.168.1.12
# Set the default PSP for EMC V-MAX to Round Robin as that is our preferred load balancing mechanism
esxcli storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_SYMM
# Enable SSH and the ESXi Shell
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell


For more deepdive information read William’s post on ESXCLI and Scripted Installs.

Version 3.1 (April, 2011)

• Logon form tab order rearranged
• Logon form will remember your last selected host / vCenter server
• On vInfo new fields Provisioned, Used and shared storage
• On vInfo new fields install Boot Required, number of Virtual Disks
• On vInfo new fields Fault Tolerance State,  FT Latency Status, FT Band width and FT Secondary Latency
• On vInfo new field 128-bit SMBIOS UUID of the virtual machine.
• On vDatastore new fields Total provisioned, Used and shared storage
• On vDatastore new fields SIOC enabled flag and congested threshold value
• On vDisk new field disk persistence mode.
• On vNetwork all IP addresses of adapter are now visible
• On vMemory new field distributed Memory Entitlement
• On vCPU new fields static Cpu Entitlement and field distributed Cpu Entitlement
• On vHost new fields Current EVC mode and Max EVC mode
• New batch command line parameters -u user and -p password
• Bugfix: custom fields not always visible on vSnapshot tab.
• Bugfix: Export to Excel, some numeric columns are saved as text instead of numbers
• RVToolsBatch.cmd with send by email example deployed in RVTools program file directory

Connect-VIServer : 8/12/2010 11:51:36 Connect-VIServer
Could not connect using the requested protocol.

I discovered that this was due to a change in the Proxy Server settings on the client. After removing this proxy address everything was back to normal.

-ray

Out-GridView : To use the Out-GridView cmdlet, install the Windows PowerShell Integrated Scripting Environment feature from Server Manager. (Could not load file or assembly ‘Microsoft.PowerShell.GraphicalHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35′ or one of its dependencies. The system cannot find the file specified.)

After some troubleshooting I discovered that this can be resolved easily by installing PowerShell ISE, which is an optional feature and requires .Net 3.5 SP1, to install this on Windows Server 2008 R2, use the Add Features Wizard or run the following Powershell command:

Import-Module ServerManager; Add-WindowsFeature PowerShell-ISE

-ray

This customer implemented a script that would run during the %firstboot section. The name of the section already explains what it is, this section will run after the reboot of the scripted install has been completed. The way this works is that ESXi creates a script in /etc/vmware/init/init.d with the prefix 999. This script will run as the last script during the boot and is generally used to configure the host. After a final reboot this script is automatically deleted and the host is ready to be added to vCenter.

The challenge however that this customer was facing is that it was using a Xsigo network environment. In this scenario a server that would need to be reinstalled would get a temporary network configuration that would only work during the first boot. Meaning that before this %firstboot section would even run the original network configuration would be restored. The problem with that however is that the original network configuration requires the drivers to be installed before it can be used. In other words, after the reboot done by the installer you will not be able to access the network unless you have loaded the drivers. This rules out downloading the drivers during the %firstboot section. Now how do we solve this?

The scripted installation has multiple sections that can contain your commands. The first section is called %post. The ESXi setup guide describes this section as follows:

Executes the specified script after package installation has been completed. If you specify multiple %post sections, they are executed in the order they appear in the installation script.

This means that in the case of this customer we will be able to use this section to download any driver package required with for instance “wget” and that is what I did. I issued the command below during the %post section and rebooted the server.

wget http://192.168.1.100/xsigo.zip

The problem however was that the package wasn’t persisted after a reboot which brought me back to right where I began, without a network after the restart. Than I figured that during the install a local datastore is created and I could use that as persistent storage. So I issued the following command in the %post section of the script:

wget http://192.168.1.100/xsigo.zip -O /vmfs/volumes/datastore1/xsigo.zip

I rebooted the installer and checked after the install if the driver bundle was there or not, and yes it was. The only thing left to do was to install the driver in the %fistboot section. This by itself is a fairly simple task:

esxupdate --bundle=/vmfs/volumes/datastore1/xsigo.zip update

As the host will need to reboot before the drivers are loaded I also added a “reboot” command at the end of to the “%firstboot” section. This ensures the drivers are loaded and the network is accessible.

I guess this demonstrates how easy a solution can be. When I first started looking into this issue and started brainstorming I was making things way too complex. By using the standard capabilities of the scripted install mechanism and a simple “wget” command you can do almost everything you need to do during the install. This also removes the need to fiddle around with injecting drivers straight into the ISO itself.

Go ESXi

Proactive DRS:

• setDRSAggressive.pl
  The script setDRSAggressive.pl sets various DRS operating parameters so that it will recommend rebalancing VMotions even when current VM demand does not make those moves appear worthwhile. As an example use case, if powerOnHosts.pl (see “Proactive DPM” posting) is used to trigger host power-ons at 8am before an expected steep increase in VM demand weekdays at 9am, setDRSAggressive.pl can also be scheduled to run at 8am to force rebalancing moves to the powered-on hosts.
• setDRSDefault.pl
  The script setDRSDefault.pl resets DRS’ operating parameters so that it resumes its normal behaviour.  (Behaviour before using setDRSAggressive.pl)
• setMaxMovesPerHost.pl
  The script setMaxMovesPerHost.pl can be used to increase DRS’ limit on the number of VMotions it will recommend in each (default every 5 minutes) regular DRS invocation

Proactive DPM:

• powerOnHosts.pl
  The script powerOnHosts.pl changes cluster settings to engender
  recommendations to power on all standby hosts and then to disable DPM so that those hosts are kept on while demand remains low.
• enableDPM.pl
  The script enableDPM.pl re-enables DPM to run in its normal reactive behavior. As an example use case, this script can be scheduled to run each weekday morning at (say) 10am (after full VM demand load is expected to be established) or at (say) 5pm (after full VM demand load is likely to diminish) to resume normal DPM operation.

I had multiple customers asking me if it was possible to schedule a change of the DRS and DPM configuration. My answer used to be yes you can script it but never managed to find a script until I bumped into these coincidentally today.

Luc Dekens is one of the leading PowerCLI script gurus and created this amazing script that creates a diagram of the relationship between VMs, VMDKs and Snapshots. Now you might wonder what the use case would be when there is a one to one relationship like the following:

Many will understand the relationship when you have a single snapshot. But is that still the case when you have multiple snapshots running on multiple disks? Probably not, check this diagram to get an idea:

Great work Luc, and my apologies for not selecting it for the Planet V12n Top 5 as it definitely deserved a spot.

So I assumed I made a mistake and went over my script. Funny thing is it just looked fine. For troubleshooting purposes I decided to strip my script and only do a “touch /tmp/test” in the %post section to see if the file would be created or not. I also removed the “automatic reboot” after the installation. When the installation was finished I went into the console and noticed my file “test” in /tmp. So I rebooted the system and checked /tmp again…. gone. HUH?

I figured it had something to do with the installer. I installed ESX manually, including a “/tmp” partition, and booted the server. I copied a bunch of random files into /tmp and rebooted the server… again the files were deleted. Now I might be going insane, but I am pretty certain this used to work just fine in the good old days ESX 3.0.X. Apparently something changed, but what?

After some googling and emailing I discovered  that this a change in behaviour is a known issue (release notes). When ESX 4.0 is booted the “/etc/init.d/vmware” cleans out /tmp. (See below) Something you might want to take into account when using /tmp.

# Clear /tmp to create more space
if IsLocalFileSystem /tmp ; then
rm -rf /tmp/*
fi

I want to thank my colleague from VMware GSS Fintan Comyns for pointing this out.

initrd=initrd.img mem=512m ksdevice=vmnic1 ip=192.168.1.123 netmask=255.255.255.0 gateway=192.168.1.1 ks=nfs://192.168.1.10:/nfs/install/ks.cfg quiet

Let’s focus on the part that’s incorrect, with ESX 3 the following bit(part of the bootstrap above) would work:

ks=nfs://192.168.1.10:/nfs/install/ks.cfg

As of ESX 4 this doesn’t work anymore, and when I do an “alt-f2″ and go to /var/log and check the esx-installer.log file it shows the following error:

mount: 192.168.1.10::nfs/install failed, reason given by server: Permission denied

After checking the permissions on my NFS share 4 times I was pretty certain that this could not cause this issue. After trying some various combinations I noticed that the format of the string for “ks” has changed. As of ESX 4 you can’t use the second colon(:) anymore. So the correct format is:

ks=nfs://192.168.1.10/nfs/install/ks.cfg

I still receive a warning but the installer does continue. If anyone knows why the following message is displayed please speak up:

No COS NICs have been added by the user

When you create a highly available environment take into account that you will need to have enough vSwitch ports available when a failover needs to occur. By default a vSwitch will be created with 56 ports and in general this is sufficient for most environments. However when two of your hosts fail in a 10 host cluster you might end up with 60 or more VMs running on a single host. If this would happen several VMs would not have a vSwitch port assigned.

The most commonly used command when creating an automated build procedure probably is:

esxcfg-vswitch -a vSwitch1

This would result in a vSwitch named “vSwitch1″ with the default amount of 56 ports. Now it is just as easy to set it up with 128 ports for instance:

esxcfg-vswitch -a vSwitch1:128

Always design for a worst case scenario. Also be aware of the overhead, some ports are reserved for internal usage. You might want to factor in some additional ports for this reason as for instance in the example above you will have 120 ports available for your VMs and not the 128 you specified.