Yellow Bricks

by Duncan Epping

VSAN Design Consideration: Booting ESXi from USB with VSAN?

· ·

** Update, as of November 21st we also support SD/USB boot with higher memory configurations when core dump partition is increased. Also read this KB article on the topic of increasing the ESXi diagnostic partition size **

One thing most probably won’t realize is that there is a design consideration with VSAN when it comes to installing ESXi. Many of you have probably been installing ESXi on USB or SD and this is also still supported with VSAN. There is one caveat however and this caveat is around the total number of GBs of memory in a host. The design consideration is fairly straight forward and also documented in the VSAN Design and Sizing Guide. Just to make it a bit easier to find it I copied/pasted it here for your convenience:

  • Use SD, USB, or hard disk devices as the installation media whenever ESXi hosts are configured with as much as 512GB memory. Minimum size for USB/SD card is 4GB, recommended 8GB.
  • Use a separate magnetic disk or solid-state disk as the installation device whenever ESXi hosts are configured with more than 512GB memory.

You may wonder what the reason is, the reason for this is that VSAN will use the core dump partition to store VSAN traces that can be used by VMware Global Support Services and the VMware Engineering team for root cause analysis when needed. So make sure when configuring host to keep this in mind when going above 512GB of memory.

Please note that this is what has been tested by VMware and will be supported, so this is not just any recommendation but could have impact on support!

Reader Interactions

Comments

  1. I understand VSAN uses part of the SSD as a write buffer. I can’t find anywhere where it explains in detail what the write buffer is. What I mean is that I understand what a write back and write through cache is. I’d like to understand how a write buffer differs from write back, and why does VSAN employ write buffer instead? Am I incorrect in assuming that with a write buffer, writes get acknowledged back to the host immediately and destaged to spinning disk asynchronously? If so, that sounds like write back. Just would like to understand the difference.

    Thanks

    • Forbsy, Its a Write Back Cache that is mirrored to two hosts and also stored on persistent flash instead of RAM. (This is also why you want to use good enterprise class flash drives like the Intel 3700 that have capacitors built in so they tolerate power loss without loosing data).

      • Thanks John. That’s interesting. I thought there might have been a difference. I wonder why VMware just doesn’t refer to it as a write back cache?

        • Thanks Duncan. I guess my confusion was comparing the terminology with PernixData. As you know that also mirrors writes to multiple hosts depending on the replication policy. PernixData refers to write back and VSAN uses write buffer. It sounds from John’s reply like they are the identical thing – just different terminology.

          • Not talking about identical solutions :). Just identical use of how writes are handled (i.e. quick acknowledgement of write back to host, replicate write, destage write to HDD).

  3. In Q3 2014 the server vendors bring 32 GB and 64 GB sd cards. Today 16 GB is no problem.

  4. Hi Duncan, what about auto deploy stateless/stateless caching, is this a good practice or recommended for VSAN ? Of course we would install a syslog and dump collector server.

    I didn’t had the chance to read the VSAN design guide yet, the answer might in it, it´s in my to do for the next weeks.

    Thansk!