Live Migration feature for bhyve

[elenamihailescu22_gmail.com]

Summary

This patch presents an implementation of the live migration feature for bhyve. The feature is a wrapper over the warm migration feature (https://reviews.freebsd.org/D28270) and sends the VM's state (memory, kernel structures, emulated devices) from a source host to a destination host over the network.

This feature sends the guest's memory while the virtual machine is still running. To do this, it uses migration rounds. In the first round, we send all the guest's memory. In the next rounds, we only send the guest's memory pages that were modified since the last round (because the guest is still running, its memory can be modified anytime and we need to update the state). Currently, we use 4 migration rounds (this may be modified in future). When the defined number of rounds are completed, the guest is stopped and the remaining memory pages alongside to the guest's kernel structures' state and emulated devices' state are copied to the destination. Then, the guest is resumed at destination.

For detecting the pages that were modified between rounds, we use a flag (VPO_VMM_DIRTY) for each vm_page. This flag is set each time vm_page->dirty field is updated and reset only when the page is migrated.

The features works only for IPv4 guests that use wired memory (-S option for bhyve). Moreover, the migration procedure works only between identical workstations.

More information regarding the implementation can be found here:

• Migrating a bhyve guest (BSDCan 2019)
• Testing and Profiling Warm And Live Migration In bhyve

The usage of the feature is detailed here: Virtual Machine Migration using bhyve.

The patch was applied over https://github.com/FreeBSD-UPB/freebsd-src/commit/d26ef5c7ac830812f07a02787f25fed5d6f8609e.

Test Plan

[jrtc27]

This patch doesn't look incremental on top of D28270 but includes (almost) all of it again. The only obvious difference I can see is that one includes BHYVE_SNAPSHOT in amd64's GENERIC, but this doesn't. I don't know if there are any security concerns with the current kernel code that warrant it being off by default, or if it's just off by default in the kernel because it's off by default in userspace.

usr.sbin/bhyve/Makefile

37

Put in the MK_BHYVE_SNAPSHOT-guarded block below?

usr.sbin/bhyve/bhyverun.c

1284–1289

Should these be made mutually exclusive?

1508

The pause and resume are shared by both restore and migration. Can that be shared, so you have:

generic preparation

if (restore) {
  restore code
}

if (migrate) {
  migrate code
}

generic resume

1579

Might be better to have a variable that means "any kind of restore"? Especially if you follow my suggestion above, you'll need two more uses of this construct otherwise.

usr.sbin/bhyve/migration.c

96

This doesn't give user-friendly error messages yet is being used in places where the errors are caused by the user's input. You're also inconsistent, sometimes using perror instead.

usr.sbin/bhyve/migration.h

57–82

Indentation for members is inconsistent

71

This seems unnecessary. You're using size_t in these so it's architecture-dependent, at which point the ABI is already forced to match (and that's implied anyway by the fact you're migrating VMs unless you're doing strange things...).

usr.sbin/bhyve/snapshot.c

35

Don't

usr.sbin/bhyve/snapshot.h

47

Why not use MAXHOSTNAMELEN? Which is 256 not 255...

52

MB is defined inside C files, don't use it here