bhyve: passthru: enable BARs before possibly mmap(2)ing them

[bz]

Summary

The first time we start bhyve with a passthru device everything is fine
as on boot we do enable BARs. If a driver (unload) inside bhyve disables
the BAR(s) as some Linux drivers do, we need to make sure we re-enable
them on next bhyve start.

If we are trying to mmap a disabled BAR for MSI-X (PCIOCBARMMAP)
the kernel will give us an EBUSY.
While we were re-enabling the BAR(s) in the current code loop
cfginit() was writing the changes out too late to the real hardware.

Move the call to init_msix_table() after the register on the real
hardware was updated. That way the kernel will be happy and the
mmap will succeed and bhyve will start.

PR: 260148
Sponsored by: The FreeBSD Foundation

Test Plan

start bhyve with iwlwifi on passthrough
kldload if_iwlwifi
kldunload if_iwlwifi
reboot
start bhyve again (fails or works again with the change)

See also PR.

[bz]

I'd love https://reviews.freebsd.org/D20623 to also go in at some point.

[markj]

Seems ok to me other than the nit, thanks.

usr.sbin/bhyve/pci_passthru.c

570

Why do we need the loop? Can this just be

if ((i = pci_msix_table_bar(sc->psc_pi)) != -1) {
    assert(i >= 0 && i <= PCI_BARMAX);
    error = init_msix_table(ctx, sc, sc->psc_bar[i].addr);

?

In fact, the third parameter to init_msix_table() is unused and could be dropped, so the BAR index i is not needed.

[bz]

Also simplify the code given the last argument to init_msix_table()
is unused we do not need to do checks for each bar. [1]

[bz]

usr.sbin/bhyve/pci_passthru.c

570

Good spot. Makes the code a lot simpler. Thank you!

[corvink]

This patch unconditionally calls init_msix_table. This breaks passthru for devices which only support MSI. I created a patch for it D33728