We had a discussion internally about ESX/ESXi caching I/Os. In particular this discussion was around caching of writes as a customer was concerned about consistency of their data. I fully understand that they are concerned and I know in the past some vendors were doing write caching however VMware does not do this for obvious reasons. Although performance is important it is worthless when your data is corrupt / inconsistent. Of course I looked around for data to back this claim up and bust this myth once and for all. I found a KB article that acknowledges this and have a quote from one of our VMFS engineers.
Source Satyam Vaghani (VMware Engineering)
ESX(i) does not cache guest OS writes. This gives a VM the same crash consistency as a physical machine: i.e. a write that was issued by the guest OS and acknowledged as successful by the hypervisor is guaranteed to be on disk at the time of acknowledgement. In other words, there is no write cache on ESX to talk about, and so disabling it is moot. So that’s one thing out of our way.Source – Knowledge Base
VMware ESX acknowledges a write or read to a guest operating system only after that write or read is acknowledged by the hardware controller to ESX. Applications running inside virtual machines on ESX are afforded the same crash consistency guarantees as applications running on physical machines or physical disk controllers.
Does it read cache? I don’t think I have ever read that it does, but sometimes in testing it seems like it is.
No it does no caching whatsoever currently. All caching mechanisms reside on the storage array side.
Another way to think of it is that since disks are disjoint between VMs there’s no sense in ESX caching the blocks; that’s an optimization left up to the guests which know more about block’s contents and usage.
The only exception is read-only linked-clone base images which could be read-cached to improve boot storm performance.
But this could be a useful feature, in some rare cases where one does not need the data to be consistent it could come in handy to have a write-cache feature pr vmdk.
Based on testing I’ve done, I believe that if you are using pvSCSI you do get some host-level read cache. May as well do read caching with whatever RAM a host has left over, its not doing anything else useful.
Thanks, Duncan.
I’m interested in the write caching as described in the original post: this may be a stupid question, but is there any caching between the guest OS and ESX (aside from the caching that the guest OS is doing itself)? In other words, if I do a fsync() in a guest app, which on a physical machine would force a flush all the way to the physical controller, will that get all the way to the physical controller on ESX as well? Or does it only get as far as some kind of virtual controller in the guest VM, which will get lost if the host suffers uncontrolled shutdown?
I’m not clear of the architectural boundaries between virtual controller and ESX, so as I say this may be a stupid question.