ESXi on Raspberry Pi - Architecting IT

Update 25 November 2020: See “Next Steps” for updated details on iSCSI configuration.

As part of VMworld 2020, VMware announced the availability of the ESXi hypervisor on a range of ARM devices, including the Raspberry Pi. Many home lab fans will be looking to get hold of this software and see how virtualising on ARM might work. Are there any practical benefits to virtualising the Raspberry Pi?

Evaluation

I currently have two “spare” Pi devices, both 4B models with 8GB of RAM. So I took the plunge and followed the instructions posted here (ESXi Arm Edition).

Getting ESXi up and running on one of the Pis was relatively easy but there are a few gotchas.

ESXi 7 requires UEFI boot and so there’s a few configuration changes needed. This includes booting from an SD card into the UEFI management screen and configuring to boot from a connected USB device (more on this in a moment). The result is that the micro-SD card slot is unusable for the O/S. Great if you have some really small micro-SD cards lying around. No so great if you expected to run a datastore on the local device.
Installation boots from a USB device to install onto another USB device. I used two memory sticks, with the running O/S on a 4GB device. VMware recommends using an externally powered USB drive or simple memory sticks as the Pi runs hot with ESXi (CPU heatsink/fan is also recommended).
Datastores can be connected via iSCSI or NFS, however I couldn’t get a USB drive to be recognised and the power draw of a USB3 SSD seemed to lock the Pi up.

Aside from the above issues (and there are more caveats in the documentation), my installation is a pretty faithful reproduction of ESXi on a server or home lab. However, ESXi itself uses about 1.3GB of RAM, leaving only around 6.5GB usable. This could support perhaps 4-5 small VMs, but would be much more useful for running containerised applications and eliminating the individual VM overhead.

Proof of Concept

Remember that Flings are just proofs of concept and not always aimed at final production. VMware suggests a range of hardware options for ARM-based workloads and this is more likely where we will see the initial development of ESXi on ARM. The challenge for end users here is to determine whether the price/performance/power/cooling ratio works better on ARM than x86. Of course applications also need to be available, but it’s not hard to find ARM-compiled versions of most open source software and operating systems.

Next Steps

Over the coming weeks I’ll be trying out some application deployments with ARM and looking at the various configuration options. One scenario I’m keen to try is iSCSI boot. It looks possible to boot ESXi on ARM from shared iSCSI storage and to use shared NFS for datastores. Performance will be terrible, but the capability offers the scenario of stateless hardware and even templated deployments by cloning the iSCSI boot LUN.

Update (25/11/2020): iSCSI Configuration is done and easy to implement!

iSCSI

To boot from an iSCSI LUN, I followed the guidelines from this post on the VMware/Arm blog. In my case, I have a QNAP appliance, on which I configured two LUNs, mapped to generic IQNs. Each of my two Pi 4Bs has a LUN each, with a separate target/LUN for each Pi. This seems to be the preferred QNAP configuration.

Remember that a micro-SD card is needed to UEFI boot the Pi. At this point the iSCSI connection can be configured. If the configuration is incorrect, on reboot the iSCSI LUN doesn’t appear in the Pi boot list.

I toyed with the idea of building a PXE environment to build the ESXi environments over the network, however that seemed a little excessive for this next test. Instead I used a USB stick configured with the ESX 7 ISO.

After validating the iSCSI LUN is available, the boot can be executed from the USB drive and installed to the iSCSI LUN. On completion, remove the USB drive and reboot from the iSCSI LUN. As per the article, the order of boot can be set to make the iSCSI LUN first.

Datastores

So with two Pis booting from network iSCSI LUNs, what about datastores? I prefer stateless machines so each Pi is configured to an NFS share also exported from the QNAP. This makes life easier to get ISO images available to the Pis. Both Pis are now connected to my vCenter Server running in the main data centre lab. Now it’s a case of seeing what these devices can do!

The Architect’s View®

We already know that ESXi runs on ARM within Project Monterey SmartNICs. With offloaded storage, networking & security, ARM for core applications could be a more compelling option, if the cost model stacks up. Although the ESXi on ARM implementation is currently a Fling, I would expect to see a fully production-baked offering released for the right combination of hardware. At this point I don’t think that will ever be the Raspberry Pi, but of course hardware advances move on rapidly, so you never know. If you can spare around $100 and some time, why not give ESXi on ARM a go and feed back your experiences?

One final thought. Using ESXi as the onboard hypervisor is great to offer a similar look and feel to traditional hardware installations. However, edge solutions don’t necessarily need the same degree of management and functionality. So I wonder if a better solution could be a custom or lightweight O/S purely for running containers. VMware could choose to do this, cutting the overhead of ESXi to just run container workloads. Now that would be interesting to see.