I was working on an automated build procedure yesterday of ESX hosts in a cloud environment. I stored my my temporary post configuration script in /tmp/ as I have been doing since 3.0.x. When the installation was finished the host rebooted and I waited on the second reboot to occur, which is part of my post configuration. Weird thing is it never happened.
So I assumed I made a mistake and went over my script. Funny thing is it just looked fine. For troubleshooting purposes I decided to strip my script and only do a “touch /tmp/test” in the %post section to see if the file would be created or not. I also removed the “automatic reboot” after the installation. When the installation was finished I went into the console and noticed my file “test” in /tmp. So I rebooted the system and checked /tmp again…. gone. HUH?
I figured it had something to do with the installer. I installed ESX manually, including a “/tmp” partition, and booted the server. I copied a bunch of random files into /tmp and rebooted the server… again the files were deleted. Now I might be going insane, but I am pretty certain this used to work just fine in the good old days ESX 3.0.X. Apparently something changed, but what?
After some googling and emailing I discovered that this a change in behaviour is a known issue (release notes). When ESX 4.0 is booted the “/etc/init.d/vmware” cleans out /tmp. (See below) Something you might want to take into account when using /tmp.
# Clear /tmp to create more space if IsLocalFileSystem /tmp ; then rm -rf /tmp/* fi
I want to thank my colleague from VMware GSS Fintan Comyns for pointing this out.
hahahah, that’s why EDA uses /root for install scripts since it supports vSphere servers 😉 too bad everybody has to find out the hard way, isn’t it? (unless they read your blog from now on of course)
It has been normal behavior for Debian/Ubuntu to delete files in /tmp on boot for some time. RHEL/CentOS uses a cron job with a default cleanup of 720 hours I think. I wonder is VMware following suit or did this just get tucked in? Or is it truly a bug?
This is standard behavior for /tmp in many Unix flavors. Use /var/tmp if you wish to retain temporary files over reboots.
I had exactly the same experience. I should have added it to my blog post about unattended ESX installations. Well done finding out why the /tmp is cleared after setup.
But ESX isn’t Linux or Unix… It is ESX. And although this might be common practice this never was the case in the previous releases.
If it is a known issue it must be an issue?! check release notes linked in the article.
It will be fixed in ESX 4.1 release.
In 3.5 there is a cronjob ‘tmpwatch’ that runs daily and removes any file in /tmp which hasn’t been used for 30 days.
ESX isn’t linux or UNIX but the service console is. It would make perfect sense for the service console to behave a lot like a linux.
At any rate, this is probably one of those things that is meant to make support a bit easier since /tmp is one of the few directories in a default install not in its own slice that grows. If someone’s ESX server falls over from / filling up (because /tmp keeps growing) good odds a reboot will fix it.
I wouldn’t say it was classified as a “known issue” because of a problem but because it is a deliberate change in behavior.
By default /tmp is not even on disk but in memory! It wouldn’t fill up your disk. Now i am okay with a change in behaviour, but why isn’t it mentioned in the documentation?
as service console is built upon rhel5 i think vmware might acknowledge the fact that user of esx and service console have a bit of Unix/Linux knowledge and so won’t put anything in /tmp.
if /tmp is in memory not on disk, it might be a good idea to really flush it, as it is well known that reboot does not really clean memory.
I’ve seen some forensic tools that can grab what’s inside the memory after a reboot.
the best way would to shutdown and plugout the power supply for several seconds, but we all know that we can afford to do this everytime we need to reboot an esx even if reboot an esx in production state would not often happen 🙂
that was my 2cents
And I am okay with that. I know why they did it, it’s just a shame it’s not documented clearly enough.
Duncan,
what do you mean by “By default /tmp is not even on disk but in memory!”?
tmp means temporary 🙂 This is normal behaviour on many UNIX flavours and BSD.