Yellow Bricks

by Duncan Epping

ESXi on ARM/Raspberry Pi for vSAN Witness purposes or for?

· ·

I was just catching up on a couple of VMworld sessions. One session that stood out to me was most definitely once again the session by Chris Wolf and Daniel Beveridge. I am not going to write up a full coverage of it, as it is mostly very similar to the session they did in the US which I posted about here.

However, what is interesting in the European edition is that Regis Duchesne comes up on stage after about 38 minutes in and he starts discussing and demoing ESXi on ARM, but more impressively ESXi on top of a Raspberry Pi. Note that these machines have very limited memory (1GB) and little CPU (64-bit SoC @ 1.4GHz) resources, and are low powered! Gotta love an intro as well that includes “been at VMware for about 20 years”.

Very interesting to see that Regis and the team managed to get ESXi booting on an RPI 3b, but also that it only uses about 500MB of the memory, which would leave room to boot one VM as Regis points out if you are lucky. One example of a use case is to use this machine as a physical vSAN Witness host for 2 host configurations. This was the immediate use case I had in mind as well for this! (Although preferably a configuration with a bit more CPU power and memory would be preferred!)

Regis also mentions the option to run 1 VM on an RPi3, but you could, of course, have multiple RPi’s running and connect them using a 1GbE switch so the VMs can communicate with each other, you could even create a cluster and move VMs between RPi’s when you are doing maintenance at the edge. Or even more VMs could potentially run on an RPi and you could use it as an IoT gateway. As Regis points out, what is great about ESXi is that it already provides isolation and QoS for VMs, which ensures that all apps running on an IoT gateway would get their fair share of resources. (Eliminate the noisy neighbor problems) Note that this is a project and very much at an alpha stage, nowhere close to being available for customers or partners, but as Regis points out… if you are a customer or partner doing things at the edge and interesting in this please let us know. The team is looking for design partners to better understand the different use case, to ensure they build something which can be useful for customers! (You can leave a comment here, let us know what you are looking to do with it and I will connect you with the right folks.)

Reader Interactions

Comments

  3. It will be very difficult to use a Raspberry Pi as Witness appliance unless the way the witness works will fundamentely change. Even if you are able to attach the necessary storage to it you still have a minimum memory requirement that currently it 8 GB even for tiny deployments.

    • Yes the engineering team understands that. 3 years ago people also said that running ESXi on ARM was impossible, but the the team has it working now on a 1GB device. All optimizations go back in to core ESXi, making it leaner and meaner than ever before. And although of course an RPi is not enterprise grade, there are various other low powered and low resource ARM devices that could be an option, that may work, running through crazy use cases like these can lead to innovation in other spaces.

      • I completely agree with that! Having ESXi available on ARM is great for Edge/IOT, and running it on the Raspberry Pi (and similar low end ARM devices) is super cool, even if it’s only good for home labs / testing / learning etc. Indeed this is driven by innovation and will drive new innovation itself. The extent of innovation is one of the key drivers that differentiates VMware from its competitors and other big players. Keep up the good work!

  4. Gosh I’ve been thinking about this for months… Please let me try it guys, my use case is my cellar…

  5. Hey Duncan, do you have any thoughts or reservations about possible performance implications from running a vSAN Witness node on a low-power device like a Pi, NAS, etc? Let’s for now ignore all aforementioned storage/memory hurdles and focus on CPU.

    I’ve been running a witness appliance on a laptop via Workstation, and it’s been quite performant. I’m now trying to see if I can take the laptop out of the picture, running a new vSAN witness appliance on a Synology DS1515+ NAS (Intel Atom C2538 2.4GHz).
    As a standalone ESXi VM, it *seems* fine so far, if noticeably a few seconds slower at basics like host info-gathering of installed vibs in VC for that host (working on devising more intensive comparisons).

    It’s on SSD storage, and ESXi largely operates in RAM after boot, so I figure it’s all CPU now. Not doing any actual vSAN stuff just yet with this test appliance.

    In short, I’m sitting here wondering, will vSAN be noticeably slower if I moved to what is very much expected to be a slower witness appliance?

    • I doubt you will notice a performance difference. We tolerate up to 500ms of latency for the witness, IO is not impacted by that latency.

      Having said that, I don’t think that using an RPi will be realistic as vSAN has certain memory requirements, but it does show what VMware is capable of doing, and that VMware is capable of optimizing the stack in such a way that it can run with very little resources.