bhyve: GPU Passthrough for dedicated AMD GPUs

[corvink]

Summary

Description:
This patch adds support for GPU Passthrough of dedicated AMD GPUs. No integrated or second dedicated GPU is neccessary to use GPU Passthrough. If no second GPU is installed, FreeBSD will run headless after booting your VM.

It's based on some other reviews:

• D26209

To simplify testing, this patch includes all changes. I'm going to rebase this patch as soon as other reviews are merged.

Prerequisites:

• CPU with VT-d or AMD-Vi

Tested Scenarios:

Hardware	OS
Intel Xeon D-1527 + AMD E8860	Windows 10	Working
Intel Xeon D-1527 + AMD E8860	Ubuntu 20.04	Working

For Windows it's only neccessary to use another BAR-emulation for passthru devices.
Linux GPU driver is searching for a VBIOS which is currently missing. This patch tries to extract VBIOS and passthrough it as PCI ROM to a guest.

How to use GPU Passthrough:

1. Use newest UEFI with enabled bus enumeration:
   • get latest edk2 sources
     • portsnap fetch extract
     • cd /usr/ports/sysutils/uefi-edk2-bhyve
     • make extract
   • modify OvmfPkg/Bhyve/BhyveX64.dsc:
     • set gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration to FALSE:
       
       gEfiMdeModulePkgTokenSpaceGuid.PcdPciDisableBusEnumeration|FALSE
   • rebuild and install new UEFI
     • cd /usr/ports/sysutils/uefi-edk2-bhyve
     • make reinstall

2. Just add your GPU to bhyve like any other pci passthru device:
   • assign ppt driver to your gpu
   • add your gpu as passthru device to your bhyve config (e.g.: -s 2,passthru,3/0/0)

3. To use a custom Video-BIOS add the path to the romfile:
   • -s 2,passthru,3/0/0,rom=<path/to/romfile>

[darkfiberiru_gmail.com]

Are there any technical limitations that make this specific to AMD GPU?

[corvink]

In D27456#613548, @darkfiberiru_gmail.com wrote:

Are there any technical limitations that make this specific to AMD GPU?

This patch extracts VBIOS only for AMD GPUs and passes it to the guest as PCI ROM.
If it's not neccessary for Nvidia GPUs to extract VBIOS and pass it to the guest, it'll work for them too. So far, I don't have a Nvidia GPU to test it.

[corvink]

• Use new UEFI provided by D27230

[khng]

usr.sbin/bhyve/pci_emul.c

1353–1354

1353–1354

Unused variables.

[corvink]

• Rebase onto main
• It's possible to load a Video-BIOS from a file now

[corvink]

• rebase

[corvink]

GPU passthrough for deciated AMD GPUs will be enabled by D33129.