vmm: Use a cpuset_t for vCPUs waiting for STARTUP IPIs.

[jhb]

Summary

Retire the boot_state member of struct vlapic and instead use a cpuset
in the VM to track vCPUs waiting for STARTUP IPIs. INIT IPIs add
vCPUs to this set, and STARTUP IPIs remove vCPUs from the set.
STARTUP IPIs are only reported to userland for vCPUs that were removed
from the set.

In particular, this permits a subsequent change to allocate vCPUs on
demand when the vCPU may not be allocated until after a STARTUP IPI is
reported to userland.

[corvink]

sys/amd64/vmm/vmm.c

2445

vm_handle_ipi is called in a rendezvous. So, the caller can already hold the rendezvous_mtx. Therefore, we need to set the MTX_RECURSE flag on mtx_init.

[jhb]

sys/amd64/vmm/io/vlapic.c

1207

Note that the call here is just after a call to vm_smp_rendezvous.

sys/amd64/vmm/vmm.c

2445

Hmm, I think vm_handle_init is the function called in a rendezvous as vm_handle_ipi calls vm_smp_redenzvous itself. I didn't get a panic when testing this in a debug kernel which normally you would get for recursing on a mutex without the flag.

[corvink]

sys/amd64/vmm/vmm.c

2445

Sry. Didn't look close enough. You're right.

2798

Shouldn't we snapshot vm->startup_cpus?

[jhb]

Snapshot the startup cpuset

[markj]

sys/amd64/vmm/vmm.c

176

Does it really need to be volatile? I can't see why. In fact, with commit 8b1adff8bcbd I think vm_snapshot_vm() will raise a warning.

[jhb]

sys/amd64/vmm/vmm.c

176

Mmm, correct, it does not as it is locked by the rendezvous_mtx.

[jhb]

Drop volatile