Enabling lockdown mode disables all direct root access to ESXi machines. Any subsequent local changes to the host must be made in a vSphere Client session or vSphere CLI command to vCenter Server using a fully editable Active Directory account. You can also use a local user account defined by the host. By default, no local user accounts exist on the ESXi system. Such accounts can only be created prior to enabling lockdown mode in a vSphere Client session directly on the ESXi system. The changes to the host are limited to the privileges granted to that user locally on that host.

I guess this table explains it a bit better, I ripped this from “it’s all virtual” so credits where credits are due:

As you can imagine the C:\ partition is not the ideal place for storing log files. I would personally recommend to use a separate drive for logfiles so avoid it from flooding any OS or Program related drives. You could pick a small size based on the expected log size and if needed increase the amount of logs that are stored and the size of the log file.

Changing this is pretty simple. Open “vpxd.cfg” and add the following line in between <log> and </log>

<directory>D:\VMware\Logs</directory>

Changing the amount of log files stored and the size is also pretty basic, in this example vCenter will store 10 logfiles which are max 10MB each:

<maxFileSize>10485760</maxFileSize>
<maxFileNum>10</maxFileNum>

Keep in mind that you will need to restart the vCenter Service after these changes before they take effect!

Default install with no redundancy:

VM Network inherits from vSwitch:

Management Network does not inherit from vSwitch:

For the default “VM Network” portgroup everything works as expected. But for the “Management Network” it doesn’t. So what’s the problem? Well it might not be a huge issue but it is something you will need to keep in mind. I wanted to add two NICs to my vSwitch0 and expected that both would be marked as “active” on the vSwitch. And this is what happens on the vSwitch, BUT the “Management Network” does not inherit the vSwitch settings so what do you think will happen? Again see the screenshot below for the details:

For some weird reason one of the vmnics is set to “unused” instead of active… Keep this in mind when installing / configuring ESXi as you might end up with less redundancy then expected. I just did a quick search if it was a known/documented change and it appears that I am not the only one who ran into this, but is does not seem to be a commonly known “issue”/change.

When Tech Support is disabled it removes the option to login to the console with “unsupported“. Please keep in mind that the console is the only way to get direct command line access to ESXi as SSH is disabled by default. This also means that in order to get access to the console you will need access to the physical host, or the IP KVM switch / DRAC / ILO for that matter. Hosts are usually located in a secured environment which removes the need for limiting console access in my opinion.

I can still imagine that people have a different opinion, but if you look at it from a support perspective you might change your mind. Troubleshooting an issue can get really complicated when there is no Tech Support access. I guess in a high secure environment you could treat ESXi as a stateless appliance and just install a new version when it fails. Personally I would prefer to find the root cause and try to prevent the same problem from occurring again.

Of course you can enable Tech Support again when needed but a reboot is required. This might cause the symptoms of the problem you were facing to disappear. It’s my recommendation to Keep Tech Support enabled.

[edit] Of course Alan “the king of powershell” Renouf jumped on this topic immediately and created a couple of lines of script which show you the current setting, disable it for all hosts or enable it for all hosts. Thanks Alan! [/edit]

Installing on a Fibre Channel SAN is supported experimentally. Do not attempt to install ESXi with a SAN attached, unless you want to try this experimental feature.

Source: ESXi Installable and vCenter Server Setup Guide (page 20)

For those interested in what “experimental” actually means, read this section on the VMware website.

I’ve tested it and removing (or renaming) the following file will lead to a blank page when the ESXi host is accessed via http(s):

/usr/lib/vmware/hostd/docroot/index.html

<edit>

William Lam created another work around which is definitely a more elegant solution: Remove the ESXi web welcome screen.

</edit>

/usr/bin/busybox mount

also might come in handy:

/usr/bin/busybox fdisk -l

Busybox… indeed, that’s what is being used under the hood and that’s what needs to be used for specific commands. Just run /usr/bin/busybox and you will see which commands you will have to your disposal. Another command I often use when working on the ESXi console is “vim-cmd”. Remember these…

I’ve been looking into creating an unattended install for ESXi. As mentioned in Lessons Learned part 2 this is not something that can be done out of the box unfortunately. I did a quick search on the internet and the VMware internal mailing lists but couldn’t find anything useful and that’s why I booted the ISO and logged into the console via “ALT-F1″ -> Unsupported -> enter.

After some fiddling around in the ESXi iso I noticed a file called “/usr/lib/vmware/installer/ThinESXInstall.py”. I am not a python guru but I guess the following lines were pretty obvious:

Steps = [ WelcomeStep, LicenseStep, TargetSelectionStep, ConfirmStep, \
WriteStep, PostConfigStep, CompleteStep, RebootStep ]

This line describes the steps taken during the install. After I noticed these I did a quick search again on the filename and an article came up of my friend from down under, Stu aka Mr Vinternals. Stu describes which steps can be removed to decrease the amount of manual intervention:

Steps = [ TargetSelectionStep, WriteStep, PostConfigStep, RebootStep ]

Only the TargetSelectionStep requires input at the moment but that is also something that can be fixed. Look at the script “ThinESXInstallSteps.py”. There is a section that describes the disk selection, you can automate it by altering it and selecting a local disk with “IsLocal()”. That’s all I can say for now….

When Jumbo Frames were introduced in ESX 3.5 a lot of people were interested but it wasn’t supported for the VMkernel which is were most people want to use it as it reduces CPU cycles used due to iSCSI/NFS traffic. When vSphere was released it was one of the things I noticed first. Full support for Jumbo Frames… however:

• Jumbo frames are not supported for VMkernel networking interfaces in ESXi. (page 54)

Although obvious in my opinion it wasn’t obvious for one of my customers:

• VMware ESXi does not support web access at this time.

Although I personally usually disable web access as it a security risk this customer had specific operational procedures around vCenter failures which included web access. It’s not a huge problem as the vSphere/vCenter Client can directly connect to an ESXi host, but it is something you will need to keep in mind when implementing ESXi.

Another thing I discovered today is that scripted/unattended installations of ESXi are not currently supported which makes deploying difficult. (Not only unsupported but also not easily set up) I am currently investigating the option to at least install it with default settings and without manual intervention… If I am successful I will post it in the next “lessons learned”.

REVISED!

Scratch!

There are two things that stood out the couple of days, on a technical level, when I was reading the ESXi installable documentation:

One of the things that used to be a requirement was the Scratch Partition. It appears that with vSphere this requirement has been removed:

During the autoconfiguration phase, a 4GB VFAT scratch partition is created if the partition is not present on another disk. When ESXi boots, the system tries to find a suitable partition on a local disk to create a scratch partition. The scratch partition is not required.

Of course this does not necessarily mean that you do not need one as explained in the second part of the paragraph:

It is used to store vm-support output, which you need when you create a support bundle. If the scratch partition is not present, vm-support output is stored in a ramdisk. This might be problematic in low-memory situations, but is not critical.

So the question remains what would my recommendation be? The answer is it depends, yes I know the easy way out. But when you have enough RAM on a host and from experience know that usually you only create support dumps on hosts which are in maintenance mode then don’t worry about it and don’t create it. However if you feel there is a need to create vm-support dumps while running production make sure there is a scratch partition with enough free space available.

Support

Yes ESXi is fully supported but there are some restrictions:

• Boot from FC SAN – Experimental Support
• Stateless PXE Boot – Experimental Support

Now what does “experimental support” mean? According to the VMware website it means the following:

VMware includes certain “experimental features” in some of our product releases. These features are there for you to test and experiment with. VMware does not expect these features to be used in a production environment. However, if you do encounter any issues with an “experimental feature”, VMware is interested in any feedback you are willing to share. Please submit a support request through the normal access methods. VMware cannot, however, commit to troubleshoot, provide workarounds or provide fixes for these “experimental features”.

So does that mean that in the case of stateless the booting process is experimental? Or the installation process in the case of boot from FC SAN?

No it does not. Everything related to ESXi is “experimental”. So what does this mean? Imagine you are facing serious storage issues and you just called VMware. VMware analyzes your environment and notices that it’s a PXE booted environment, they will more than likely give your support call a lower priority. Not only a lower priority but the support is “best effort”, no guarantees.