bhyve: Split vmexit handling into a separate file

[markj]

Summary

Put it in amd64, since most of it is MD and won't be used on arm64. Add
a bit of glue to bhyverun.h to make CPU startup and shutdown work
without having to export more global variables. AP startup will be
reworked further in a future revision.

This makes bhyverun.c much more machine-independent.

No functional change intended.

[jhb]

Like the commit in general.

usr.sbin/bhyve/bhyverun.c

485

It might be useful to move the logic from vmexit_suspend up into fbsdrun_deletecpu with the new signature, etc. as a commit prior to this one (so people can bisect it just in case) and then this commit is a bit simpler as it's just moving the vmexit handler. Also, looking at the new code, I almost wonder if you don't want something like:

pthread_mutex_lock(&resetcpu_mtx);
if (!CPU_ISSET(...)) {
}

CPU_CLR(vcpu, &cpumask);

if (vcpu != BSP) {
    pthread_cond_signal(...);
    pthread_mutex_unlock(...);
    pthread_exit(...);
}

while (!CPU_EMPTY(&cpumask))
    pthread_cond_wait(...);
pthread_mutex_unlock(...);

Could even keep the lock as a file-level global and use it instead of atomics in fbsdrun_addcpu if you wanted.

1028–1029

We just set these always now it seems? Wonder if it might be worth it to do that as a standalone commit first just so there's less noise in this commit?

[corvink]

LGTM but I prefer to split this commit as jhb@ already mentioned.

usr.sbin/bhyve/bhyverun.h

54

Nit: This is vmexit related. So, IMHO it should be defined in vmexit.h.

[markj]

usr.sbin/bhyve/bhyverun.c

485

https://reviews.freebsd.org/D40573

1028–1029

Yes, sorry, I neglected to mention this. https://reviews.freebsd.org/D40572

[markj]

Rebase, handle review feedback.