by Duncan Epping
Just dropped the following short script in a topic on the VMTN Community. It shuts down every VM on the host and than the host itself, but it will enter maintenance mode first, just to be sure that no other vm’s get migrated to this host.
VMLIST=’vmware-cmd -l’
for config in ${VMLIST}
do
vmware-cmd $config stop trysoft
done
vimsh -n -e /hostsvc/maintenance_mode_enter
shutdown -h now
addition: thanks forbes for the trysoft addition.
At a customer site we noticed that the ESX hosts were swapping, Nagios generated a nice alarm. After some research it seemed like certain VM’s were swapping to the VMFS volume, so not inside the OS but VMware swap usage. A closer look at the system revealed that we weren’t overcommitting. There was over 6GB of memory free and there were no limit’s set to the specific VM. Could it be just Nagios or… No, esxtop with the following commands “s2
The column swcur displays the current swap file usage, I marked the values higher than 0 red.
After a couple of searches it seemed that there is little info about swcur. But Kit Colbert, a VMware employee, posted on the vmtn forum about checking your current memory / swap usage in the file “/proc/vmware/sched/mem”. With cat you can easily display this, and with “watch -n 1” you can refresh your view every second. The following output was retrieved via the command “watch -n 1 cat /proc/vmware/sched/mem”:
We’ve migrated a VM which was swapping according to esxtop and nagios to another host, and as expected the swap remained. We powered down a VM that was swapping, and although the host had more than enough free mem available, the swap returned. It was less than before but still… The funny thing is that according to Kit it’s all about the column “swap out” and we did not see much action going on there.
I’m dazzled, anyone?
A while back I blogged about the customization specification wizard failing when entering the password. Today I visited the same customer again. This problem only occurred when running it from the VirtualCenter itself. Today I upgraded the VirtualCenter server to 2.5 Update 1 and the problem is solved… Still don’t know why it happened,
I just encountered the same bug that Aaron Delp blogged about in January:
It appears we have found a possible bug in the Deploy from Template Command in ESX 3.5. When you create a Windows Server based template and then try to deploy directly into an Active Directory with customization, the new system will get an error that a service failed to start when the machine is launched. This is because the VMWare BootRun service is not removing itself properly after deployment. This does not happen with deployments into a workgroup.
If you aren’t familiar with the BootRun service, this service will make all of the customizations after the sysprep work is complete during the deployment. You usually never see it because it runs on the first boot, makes the changes, and then removes itself from the machine.
In this case, the files are removed but the service entry is still there, hence the error that it can’t start up. VMWare has confirmed this to be a problem and they are investigating.
This customer runs the latest and greatest version of ESX and VirtualCenter so it appears that the bug hasn’t been fixed yet.
Monday evening a colleague changed the ip-address of three VMware ESX hosts. He followed the standard VMware procedure, which usually works like a charm. In this case after the ip-address was changed HA did not work anymore. Disabling and enabling the HA resulted in the following error: “Configuration of host IP address is inconsistent on host …”
After a close inspection the following error was found in /var/log/vmware/vpx-rupgrade.log:
VMwareerrortext=ft_gethostbyname and hostname -i return different addresses: 10.21.10.81, 10.21.5.12 and 10.21.1.21
The command “hostname -i” resulted in the following:
[root@bla-01 /var/log/vmware]# hostname -i
10.21.1.21
The command “ft_gethostbyname” returned the following:
[root@bla-01 /opt/vmware/aam/bin]# ./ft_gethostbyname
10.21.10.81 bla-01
10.21.5.12 bla-01
So for some reason ESX resolved the wrong address. The hosts file wasn’t the problem, but FT_HOSTS which is automatically generated by the AAM Client(High Availability) was:
[root@bla-01 /etc]# more FT_HOSTS
# Auto-generated FT_HOSTS file. Timestamp: Mon Jun 2 19:05:09 2008
10.21.10.81 bla-01
10.21.5.12 bla-01
10.21.10.82 bla-02
10.21.5.14 bla-02
10.21.10.83 bla-03
10.21.5.16 bla-03
So I moved the FT_HOSTS to FT_HOSTS.BAK:
[root@bla-01 /etc]# mv FT_HOSTS FT_HOSTS.BAK
Reconfigured the cluster for HA and everything works like expected again:
[root@bla-01 /etc]# more FT_HOSTS
# Auto-generated FT_HOSTS file. Timestamp: Wed Jun 4 10:39:52 2008
10.21.1.21 bla-01
10.21.5.12 bla-01
10.21.1.22 bla-02
10.21.5.14 bla-02
10.21.1.23 bla-03
10.21.5.16 bla-03
Deleting the cluster, removing the hosts from the cluster and or reconfiguring HA did not once update the FT_HOSTS file. I would expect that with every “reconfigure for HA” action an update or check of the FT_HOSTS file would be done.