bhyve on FreeBSD and VM live migration: quo vadis?

When I think about bhyve Live Migration, it’s something I encounter almost daily in my consulting calls. VMware’s struggles with Broadcom’s licensing issues have been a frequent topic, even as we approach the end of 2024. It’s surprising that many customers still feel uncertain about how to navigate this mess. While VMware has been a mainstay in enterprise environments for years, these ongoing issues make customers nervous. And they should be – it’s hard to rely on something when even the licensing situation feels volatile.

Now, as much as I’m a die-hard FreeBSD fan, I have to admit that FreeBSD still falls short when it comes to virtualization – at least from an enterprise perspective. In these environments, it’s not just about running a VM; it’s about having the flexibility and capabilities to manage workloads without interruption. Years ago, open-source solutions like KVM (e.g., Proxmox) and Xen (e.g., XCP-ng) introduced features like live migration, where you can move VMs between hosts with zero downtime. Even more recently, solutions like SUSE Harvester (utilizing KubeVirt for running VMs) have shown that this is now an essential part of any virtualization ecosystem.
↫ gyptazy

FreeBSD has bhyve, but the part where it falls short, according to gyptazy, is the tool’s lack of live migration. While competitors and alternatives allow for virtual machines to be migrated without downtime, bhyve users still need to shut down their VMs, interrupt all connections, and thus experience a period of downtime before everything is back up and running again. This is simply not acceptable in most enterprise environments, and as such, bhyve is not an option for most users of that type.

Luckily for enterprise FreeBSD users, things are improving. Live migration of bhyve virtual machines is being worked on, and basic live migration is now supported, but with limitations. For instance, only virtual machines with a maximum of 3GB could be migrated live, but that limit has been raised in recent years to 13 to 14GB, which is a lot more palatable. There are also some issues with memory corruption, as well as some other issues. Still, it’s a massive feat to have live migration at all, and it seems to be improving every year.

The linked article goes into much greater detail about where things stand, so if you’re interested in keeping up with the latest progress regarding bhyve’s live migration capabilities, it’s a great place to start.