bhyve: Optionally put vCPUs back in the debug state after resuming

[markj]

Summary

When the gdb stub is configured to pause guest execution upon boot (i.e.,
the "w" flag is passed to -G), vCPUs end up suspended in two senses: first,
suspended by the GDB stub (marked in the vcpus_suspended set), and suspended
by the kernel (because fbsdrun_addcpu() suspends APs before spawning their
vCPU threads). When the guest is resumed by the debugger, vCPUs are
unsuspended in both senses, but this is not correct for APs.

Hack around this problem by re-suspending vCPUs after the debugger resumes
guest execution, if they were suspended beforehand.

[jhb]

Hmm, I guess I'm not sure why the vCPU would already be in the debug_cpus set if it hasn't done anything yet? This would seem like a bug where we suspended a vCPU that wasn't in vcpus_active yet?

[markj]

In D46196#1053211, @jhb wrote:

Hmm, I guess I'm not sure why the vCPU would already be in the debug_cpus set if it hasn't done anything yet? This would seem like a bug where we suspended a vCPU that wasn't in vcpus_active yet?

The terminology is inconsistent, which makes this confusing. fbsdrun_addcpu() calls vm_suspend_cpu() before starting the vCPU thread, and the VM_SUSPEND_CPU ioctl puts it in the debug_cpus set. It would be nice to detangle this a bit, but for now my aim is merely to make sure that APs behave the same whether or not vCPUs block in gdb_cpu_add().

[corvink]

In D46196#1053211, @jhb wrote:

Hmm, I guess I'm not sure why the vCPU would already be in the debug_cpus set if it hasn't done anything yet? This would seem like a bug where we suspended a vCPU that wasn't in vcpus_active yet?

When sending INIT to a vCPU it'll be suspended too. It's uncommon but possible to restart an already running CPU by sending INIT to it. In this situation you'll always end up with a suspended vCPU that was previously active.

[jhb]

usr.sbin/bhyve/gdb.c

940

So then, isn't this case now always true, or at least always true for APs?

959

And isn't this also always true once you have passed the first if-clause above?

[jhb]

usr.sbin/bhyve/gdb.c

940

I guess this is only true if GDB is attached while the vCPU thread starts up. This used to happen at the time of startup IPI, now it only happens for the "wait" case.

954

959

But this is I think always true for APs.

I was hoping we could refactor this a bit to be simpler since we no longer have to deal with vCPUs arriving while running. Perhaps though it would help if some of the other comments were updated (e.g. the comment before the first if perhaps should now say that it only applies to the case where execution is paused while waiting for a debugger, or while the debugger has attached but hasn't yet resumed the guest).

[markj]

usr.sbin/bhyve/gdb.c

940

Yes, it's confusing. vcpus_suspended is just recording those vCPUs which are suspended because the stub is waiting for a debugger to attach. It's unrelated to the kernel's notion of suspended vCPUs.

959

Yes, this is currently always true for APs, I am just trying to be defensive here in case the implementation of AP startup changes again.