I noticed a couple of people reported this problem in the last two months so I figured a blog post would be useful. This thread on VMTN triggered this article. If your ESXi host is disconnected from vCenter (even 5.0 and 5.1 appear to be impacted by this) and you see error messages in your log files about free space like these:
WARNING: VisorFSObj: xxxx: Cannot create file /var/spool/snmp/xxxxxxxx_x_
x_xxxx.trp for process hostd-worker because the inode table of its ramdisk (root) is full.
VmkCtl Locking (/etc/vmware/esx.conf) : Unable to create or open a LOCK file. Failed with reason: No space left on device
This could be caused by the fact that ESXi is running out of inodes. You can simply check that on the command line by using the following command:
stat -f /
The outcome of this command will look as follows:
File: “/”
ID: 1 Namelen: 127 Type: visorfs
Block size: 4096
Blocks: Total: 449852 Free: 324368 Available: 324368
Inodes: Total: 8192 Free: 55
As you can see the amount of “free” inodes is low and this is causing the experienced issues. In some cases it is reported (by vdsyn in this case) that “/var/spool/snmp/” is full and needs to be cleaned out. In this KB Article “/var/run/sfcb/” is explicitly called out and also explains what you can delete and how. So make sure to look at those two directories when an ESXi host is disconnected from vCenter.
Hi Epping,
you can also use command df -i to check the inode useage of the filesystem.
In the *nix environment, the inode allocation is based on the filesystem.
Euuh, there is no “df -i” on ESXi?
Hi,
as you are on rare problems with esxi, I have a problem on several servers, not being able to connect via ssh to the esxi console. Problem is, I almost find no reference to it in vmware’s forum (strangly some references in centOS forums and at HP, not related to esx at all). Here is what happens:
ssh yourserver.com
Server refused to allocate pty
you can login with
ssh -T yourserver.com
though (you don’t get a prompt, but can issue commands and receive the results)
I suspect somehow, that this may be related to putty and strange hang ups I got, when using putty with a management tool (http://www.visionapp.com/germany/solutions/asg-remote-desktop.html) which I just recently solved by specifying http://screencast.com/t/veLMANQM
what is annoying is, that you can’t workaround this problem without rebooting the esxi host..
If you got some time on hand, please have a look into it. Contact me, if you need more information or access to an affected system.
I’ve got two hosts that are running VM’s and have disconnected from vCenter.
I also get the Server refused to allocate pty, and have worked around SSH by using winscp and issue commands using the terminal.
I have attempted to restart the management agent (services.sh restart), and forcefully kill/restart the hostd service.
I do not appear to be running out of inodes.
Just like the users who have posted in this thread http://communities.vmware.com/message/2071902, I had a change in paths, and have issues a rescan prior to the hosts disconnecting.
I am looking for a way to restart the management agents successfulyl without rebooting the hosts.
In esi just type
services.sh restart
As long as (and its not set this way by default) as your VM’s are not set “Start/Stop with the host” then your VM’s will be unaffected.