Since its initial release, CRC has been running on macOS as well as on Windows and Linux. However macOS support was limited to Apple machines with an Intel CPUs until recently. With Apple’s ongoing switch to aarch64 M1 CPUs, we’ve been asked numerous times when CRC would get M1 support. This has changed with the recently released crc 2.4.1. This blog post will describe the challenges we had to overcome when we decided to add that support.

Background

CRC is a tool which allows you to run an OpenShift cluster or podman containers on a laptop or local workstation. It can be used on a macOS, Windows or Linux machine. It runs its OpenShift cluster/podman runtime by starting a virtual machine (VM) using a disk image built in advance and packaged in what we call a bundle. On each platform, we try to use native hypervisors.

On macOS, crc uses hyperkit as its hypervisor. It has served us well over the years, but lately it hasn’t seen a lot of maintainance, and we’ve been hitting a few annoying bugs in its ocaml qcow2 implementation.

However, the main problem with hyperkit is that it’s x86_64 only. It does not provide any M1 support, so we had to look for alternatives if we want to run CRC on a M1 mac.

QEMU and virtualization.framework

When looking for hypervisors we could use on macOS, we found two candidates:

• QEMU with its hvf acceleration
• Apple’s own virtualization framework

QEMU

QEMU is a widely used hypervisor on Linux systems, but it can also be used on Windows and macOS. On macOS, it can use Apple’s hypervisor framework to make use of virtualization support in hardware. It has a huge range of features. In particular it supports many virtio devices including virtiofs for file sharing, Its main drawback is that for supply chain trust reasons, we’d have to maintain our own QEMU builds, which can be quite an endeavour given the size of QEMU’s code base (2 million lines of C code).

Apple’s virtualization framework

This is a high level Objective-C/swift framework to start virtual machines on Apple hardware with a small set of virtio devices (disk, network, …). It was introduced in macOS ‘Big Sur’ 11.0, and only supports macOS and Linux virtual machines. Luckily, our bundle is a Linux image, so it’s good enough for us! It obviously supports both x86_64 and M1 CPUs.

Its main advantage and drawback is that it’s a high-level framework maintained by Apple. This means very little code is needed to create a virtual machine with it, and Apple maintains and updates all the low-level virtualization code for us. This is also a drawback, as this framework is closed source. This means we fully depend on Apple for fixing any bugs that we might find. And unfortunately, we’ve already hit such bugs in Apple’s implementation of virtiofs, a bug has been filed, but we have no control over when it will get fixed, if ever.

Introducing vfkit

Apple’s virtualization framework looks promising, but it’s just a framework. A framework is similar to a shared library on Linux, your code can link to it to use what it offers, but you can’t directly use it as a commandline or GUI tool. One particular caveat with the virtualization framework is that any virtual machines you start with it will only stay alive as long as the process which started it is alive.

We could not find any pre-existing FOSS commandline tool to manage virtual machines with Apple’s virtualization framework, but luckily for us, there are go bindings for it. This allowed us to write a small commandline wrapper using these bindings which we called vfkit It does not cover all features exposed by Code-Hex/vz and the virtualization framework, but we can expand this over time. At the moment, its commandline is enough for CRC’s needs, including support for both system and usermode networking.

Where we’re at

With vfkit, we have a way of starting virtual machines on M1 hardware. The second thing that we need is a bundle with the appropriate disk images. The two main changes we had to do were:

• move from qcow2 to raw disk images as this is the only format Apple’s virtualization framework supports. This meant modifying CRC to make use of sparse files as much as possible to minimize disk usage. The clonefile system call was also very handy to limit the amount of disk space that we use
• the aarch64 kernel must be uncompressed, otherwise the virtual machine won’t start. This is fairly tedious to figure out as you don’t get any output from the virtual machine to let you know what went wrong.

After these changes, we were able to generate a podman bundle to use natively on a M1 machine. As its x86_64 counterpart, this bundle is built on top of Fedora CoreOS. Running OpenShift on a M1 CPU won’t be possible soon, as it’s currently based on Red Hat CoreOS 8, and its kernel is incompatible with M1 CPUs.

vfkit support for x86_64 hardware landed in crc 2.3.0 release, and so far did not have many issues reported. We merged M1 vfkit support in crc 2.4.1, but had last minute issues related to signing, so it’s only available as an unsigned installer for this release. Hopefully this will be solved in time for crc 2.5.

Once this is finalized, we still have plenty to explore in vfkit and M1 support: file sharing support, OKD bundles, microshift bundles, … so stay tuned!

As usual, we’d like to get as much feedback as possible on all this work!