vmm: do not call mtx_lock in icr_low write handler

[corvink]

Summary

A x2apic access is handled by a wrmsr exit. This handler is called in a
critical section. So, we can't lock a mtx in the icr_low handler.

Reported by: kp
Fixes: c0f35dbf19c3c8825bd2b321d8efd582807d1940

[jhb]

Alternatively, could just add a new spin mutex lock for startup_cpus. (That's what I had planned to do). This isn't a critical performance path such that atomic ops is a big win over just having a small lock for this.

@pho had reported this to me via e-mail as well.

[pho]

D37452.id113378.diff fixed the issue for me.

[markj]

In D37452#851606, @jhb wrote:

Alternatively, could just add a new spin mutex lock for startup_cpus. (That's what I had planned to do). This isn't a critical performance path such that atomic ops is a big win over just having a small lock for this.

I think that makes more sense. When CPU scalability isn't a major concern, it's almost always better to rely on mutexes for synchronization.

As an alternative to using a separate lock, I wonder if it can make sense to have vlapic_icrlo_write_handler() trigger a vmexit that gets handled in vm_run() instead? We can lock a regular mutex there.

[corvink]

• handle setting of startup_cpus in a vm_run handler

[corvink]

• use CPU_COPY to set ipimask in SIPI case

[corvink]

sys/amd64/vmm/io/vlapic.c

1189

This wasn't done in the !vlapic->ipi_exit case before. I don't think that resetting the vlapic is an issue for old bhyve binaries. So, I didn't masked it to keep the code cleaner.

If you complain about it, I'm going to mask it.

[jhb]

LGTM with the suggested fix.

sys/amd64/vmm/io/vlapic.c

1215

[corvink]

• set correct cpuid for STARTUP_AP exits