GVT-d support for bhyve

[corvink]

Summary

Commit message:

bhyve: pass address of OpRegion to the guest

Don't allow access to the physical ASLS register. It contains a host
address which is meaningless for the guest. Additionally, it allows the
guest to safely rewrite this register.

This is the last commit required for GVT-d. Nevertheless, it might not
work due to missing firmware support.

Description:
Adds support for Intels GVT-d feature to passthrough an integrated graphics device to a bhyve-guest.
No dedicated graphic card is neccessary to use this feature. If there is no dedicated graphic card build in, the FreeBSD host runs headless after booting the guest.

This patch is split into mutliple commits (see https://github.com/Beckhoff/freebsd-src/commits/phab/corvink/gvt-d). I'm going to create multiple patches to merge it. As soon as some patches are merged, this patch will be rebased.
Depending patches:

• D28280
• D31578
• D33010
• D33011
• D33129
• D33161
• D33188

Prerequisites:

• Intel CPU with VT-d
• BIOS option "PM Support" set to Enabled

How to enable GVT-d:
Use an updated OVMF (see below at Installation steps).

Just add your integrated graphics device to bhyve as any other PCI Passthru device.
If you add a GOP driver as ROM file to your GPU, you won't require an OS driver for display output.

bhyve -c 2 -m 4G -A -H -S -w \
  -s 0,hostbridge \
  -s 2,passthru,0/2/0[,rom=<path/to/gop.rom>] \
  -s 4,virtio-blk,/root/win/win10.img \
  -s 5,virtio-net,tap10 \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_CODE.fd \
  win10

Tested Scenarios:

OS
Windows	Working
Debian	Working
Ubuntu 20.04	Working
FreeBSD	Working

Tested with Windows as a guest:

Architecture
Sandy Bridge	Not Working (Windows reports that the device works properly but there's no correct display output)
Ivy Bridge	Working
Haswell	Working
Kabylake	Working
Coffee Lake	Working

Installation steps:

Update your OVMF:

GVT-d_v1.diff43 KBDownload

For those who don't wanna rebuild OVMF:

BHYVE_CODE.fd3 MBDownload

You have to option to use GPU passthrough:

1. Running a VM with GOP driver
   1. Add a GOP driver as ROM file to your GPU
   2. Enjoy GPU passthrough
2. Running a VM without GOP driver
   1. Install your VM on the "old" way with GVT-d disabled
   2. Boot into your VM with GVT-d enabled and install the graphics driver
   3. Boot your VM with GVT-d enabled
   4. Enjoy GPU passthrough

[corvink]

Add copyright header to new file "pci_igd_lpc.c"

[corvink]

Fixed Issues:

1. Do not patch cfgread and cfgwrite of passthru devices:

The device emulation functions are shared by every passthru device.
If igd emulationen patches it's cfgread and cfgwrite functions, it will be changed for every passthru device too.
To avoid this, add a passthru_type to decide which function to take.

[corvink]

Updates:

• Use an own device emulation for GVT-d
• Do not modify the current implementation of passthru devices

[khng]

I am curious is it possible you split out mmio remap handling to another differential review? This may make the mmio remap handling part quicker to be reviewed and merged.

[grehan]

D24066 would cover the unmap code: this work could reuse that.

[corvink]

Updates:

• Reuse D24066

[khng]

In D26209#594999, @c.koehne_beckhoff.com wrote:

Updates:

• Reuse D24066

In D26209#594999, @c.koehne_beckhoff.com wrote:

Updates:

• Reuse D24066

I think I need to make sure whether it is caused by D24066 or not:

Tracing pid 172 tid 100090 td 0xfffffe005ee35500
kdb_enter() at kdb_enter+0x37/frame 0xfffffe005f608fe0
vpanic() at vpanic+0x19e/frame 0xfffffe005f609030
panic() at panic+0x43/frame 0xfffffe005f609090
trap_fatal() at trap_fatal+0x387/frame 0xfffffe005f6090f0
trap_pfault() at trap_pfault+0x97/frame 0xfffffe005f609150
trap() at trap+0x2ab/frame 0xfffffe005f609260
calltrap() at calltrap+0x8/frame 0xfffffe005f609260
--- trap 0xc, rip = 0xffffffff82b8fc3b, rsp = 0xfffffe005f609330, rbp = 0xfffffe005f609370 ---
intel_uncore_init_mmio() at intel_uncore_init_mmio+0xdb/frame 0xfffffe005f609370
i915_driver_probe() at i915_driver_probe+0x638/frame 0xfffffe005f609410
i915_pci_probe() at i915_pci_probe+0x5d/frame 0xfffffe005f609460
linux_pci_attach_device() at linux_pci_attach_device+0x56f/frame 0xfffffe005f6094c0
device_attach() at device_attach+0x3ca/frame 0xfffffe005f609500
device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xfffffe005f609530
bus_generic_driver_added() at bus_generic_driver_added+0x58/frame 0xfffffe005f609550
devclass_driver_added() at devclass_driver_added+0x39/frame 0xfffffe005f609590
devclass_add_driver() at devclass_add_driver+0x147/frame 0xfffffe005f6095d0
_linux_pci_register_driver() at _linux_pci_register_driver+0xcf/frame 0xfffffe005f609600
i915kms_evh() at i915kms_evh+0x39/frame 0xfffffe005f609610
module_register_init() at module_register_init+0xbd/frame 0xfffffe005f609640
linker_load_module() at linker_load_module+0xbf1/frame 0xfffffe005f609960
kern_kldload() at kern_kldload+0xe6/frame 0xfffffe005f6099a0
sys_kldload() at sys_kldload+0x5b/frame 0xfffffe005f6099d0
amd64_syscall() at amd64_syscall+0x135/frame 0xfffffe005f609af0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe005f609af0
--- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x800383eea, rsp = 0x7fffffffe6d8, rbp = 0x7fffffffec50 ---

EDIT: Oh my own mistake of KBI mismatch

[grehan]

D24066 should be committed soon so one less obstacle for this work.

[wanpengqian_gmail.com]

There is some code style should be fixed.

From style(9) -- kernel source file style guide

/* Most single-line comments look like this. */

[corvink]

Fix code style of comments

[corvink]

usr.sbin/bhyve/pci_emul.h

313 ↗

(On Diff #79652)

Changing this function would enable GPU-Passthrough of dedicated graphics to a Windows VM:

static __inline int
is_passthru(struct pci_devinst *pi)
{
    return ((strcmp(pi->pi_d->pe_emu, "passthru") == 0) ||
            (strcmp(pi->pi_d->pe_emu, "gvt-d") == 0));
}

[corvink]

Updates:

• Implement GVT-d emulation as passthru device.
• Include D24066 for easier testing (will be removed as soon as it's merged)

[corvink]

• Works with new OVMF (see D27230) now

[olevole_olevole.ru]

Broken for FreeBSD 13.0-ALPHA2:

--- all_subdir_usr.sbin ---
        uint64_t cpu_maxphysaddr, pci_emul_memresv64;
                                  ^
/usr/jails/src/src_13/src/usr.sbin/bhyve/pci_emul.c:1256:8: error: unused variable 'regs' [-Werror,-Wunused-variable]
        u_int regs[4];
              ^
/usr/jails/src/src_13/src/usr.sbin/bhyve/pci_emul.c:1255:11: error: unused variable 'cpu_maxphysaddr' [-Werror,-Wunused-variable]
        uint64_t cpu_maxphysaddr, pci_emul_memresv64;
                 ^
3 errors generated.

[corvink]

• fix compiling

[olevole_olevole.ru]

In D26209#633596, @c.koehne_beckhoff.com wrote:

• fix compiling

Stilll not built for 13.0-RCx:

/usr/src/sys/amd64/vmm/io/ppt.c:477:32: error: too few arguments to function call, expected 5, have 3
        ppt = ppt_find(bus, slot, func);

I suspect for ppt_unmap_mmio:

error = ppt_find(vm, bus, slot, func, &ppt);
if (error)
        return(error);

?

[khng]

In D26209#596955, @khng300_gmail.com wrote:

In D26209#594999, @c.koehne_beckhoff.com wrote:

Updates:

• Reuse D24066

In D26209#594999, @c.koehne_beckhoff.com wrote:

Updates:

• Reuse D24066

I think I need to make sure whether it is caused by D24066 or not:

Tracing pid 172 tid 100090 td 0xfffffe005ee35500
kdb_enter() at kdb_enter+0x37/frame 0xfffffe005f608fe0
vpanic() at vpanic+0x19e/frame 0xfffffe005f609030
panic() at panic+0x43/frame 0xfffffe005f609090
trap_fatal() at trap_fatal+0x387/frame 0xfffffe005f6090f0
trap_pfault() at trap_pfault+0x97/frame 0xfffffe005f609150
trap() at trap+0x2ab/frame 0xfffffe005f609260
calltrap() at calltrap+0x8/frame 0xfffffe005f609260
--- trap 0xc, rip = 0xffffffff82b8fc3b, rsp = 0xfffffe005f609330, rbp = 0xfffffe005f609370 ---
intel_uncore_init_mmio() at intel_uncore_init_mmio+0xdb/frame 0xfffffe005f609370
i915_driver_probe() at i915_driver_probe+0x638/frame 0xfffffe005f609410
i915_pci_probe() at i915_pci_probe+0x5d/frame 0xfffffe005f609460
linux_pci_attach_device() at linux_pci_attach_device+0x56f/frame 0xfffffe005f6094c0
device_attach() at device_attach+0x3ca/frame 0xfffffe005f609500
device_probe_and_attach() at device_probe_and_attach+0x70/frame 0xfffffe005f609530
bus_generic_driver_added() at bus_generic_driver_added+0x58/frame 0xfffffe005f609550
devclass_driver_added() at devclass_driver_added+0x39/frame 0xfffffe005f609590
devclass_add_driver() at devclass_add_driver+0x147/frame 0xfffffe005f6095d0
_linux_pci_register_driver() at _linux_pci_register_driver+0xcf/frame 0xfffffe005f609600
i915kms_evh() at i915kms_evh+0x39/frame 0xfffffe005f609610
module_register_init() at module_register_init+0xbd/frame 0xfffffe005f609640
linker_load_module() at linker_load_module+0xbf1/frame 0xfffffe005f609960
kern_kldload() at kern_kldload+0xe6/frame 0xfffffe005f6099a0
sys_kldload() at sys_kldload+0x5b/frame 0xfffffe005f6099d0
amd64_syscall() at amd64_syscall+0x135/frame 0xfffffe005f609af0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe005f609af0
--- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x800383eea, rsp = 0x7fffffffe6d8, rbp = 0x7fffffffec50 ---

EDIT: Seems not.

In D26209#656624, @olevole_olevole.ru wrote:

In D26209#633596, @c.koehne_beckhoff.com wrote:

• fix compiling

Stilll not built for 13.0-RCx:

/usr/src/sys/amd64/vmm/io/ppt.c:477:32: error: too few arguments to function call, expected 5, have 3
        ppt = ppt_find(bus, slot, func);

I suspect for ppt_unmap_mmio:

error = ppt_find(vm, bus, slot, func, &ppt);
if (error)
        return(error);

?

Should be buildable by updating https://reviews.freebsd.org/D24066 .

[olevole_olevole.ru]

In D26209#656625, @khng wrote:

Should be buildable by updating https://reviews.freebsd.org/D24066 .

You are right, i missed this patch, thx

[corvink]

I'm going to rebase this patch soon.

[khng]

In D26209#656995, @c.koehne_beckhoff.com wrote:

I'm going to rebase this patch soon.

The change is upstreamed. You could integrate it with main without integrating your own D24066.

[corvink]

In D26209#656996, @khng wrote:

In D26209#656995, @c.koehne_beckhoff.com wrote:

I'm going to rebase this patch soon.

The change is upstreamed. You could integrate it with main without integrating your own D24066.

I've planned to rebase onto main. D26035 is also upstreamed. It will change this patch a bit too.

[corvink]

• rebase

[corvink]

• do not allocate Graphics Stolen Memory inside PCI Region
  
  This avoid overlapping of GSM and BARs. GSM is located at Top of Low Usable RAM now.

[khng]

usr.sbin/bhyve/pci_emul.c

860 ↗

(On Diff #88259)

[corvink]

I got some new information about Intel Graphics Devices. GOP driver is working properly on my system now. It requires some firmware patches unless it's possible to create and use an E820 table by bhyve. I'm going to update this patch soon.

[corvink]

• GOP driver is working now

Note: The patch is based on 13.0 not 14.0-CURRENT.

[corvink]

• rebase on 14.0-CURRENT

[corvink]

• rebase on CURRENT

[wanpengqian_gmail.com]

I got a intel Xe DG1 dedicated graphics adapter. I will try it later.

[corvink]

In D26209#788649, @wanpengqian_gmail.com wrote:

I got a intel Xe DG1 dedicated graphics adapter. I will try it later.

Main issue of GVT-d is that Intels integrated graphic devices are using non-standard PCI resources. That's ok for integrated graphics because there's only a limited number of mainboard they can be used on. So, the BIOS can include some special drivers for handling those resources.

For dedicated graphics the situation is different. They can be used on any system. So, they should be self containing. Otherwise, the dedicated graphic can't work on older systems (because they don't have a chance to include special drivers).

I don't know if the dedicated graphic requires a ROM or not. 14.0-CURRENT supports a ROM for passthru devices (see D33129) but you have to use a patched OVMF.

[marietto2008_gmail.com]

Hello.

I'm running FreeBSD 13.1-RELEASE and I'm trying to passthru my CoffeeLake-S GT2 [UHD Graphics 630] on a Linux VM but it failed. This graphic card :

ppt4@pci0:0:2:0: class=0x038000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x3e98 subvendor=0x1458 subdevice=0xd000
vendor = 'Intel Corporation'
device = 'CoffeeLake-S GT2 [UHD Graphics 630]'
class = display

below there are the bhyve parameters that I've used :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,nvme,/dev/$vmdisk0,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk2 \
-s 3,ahci-hd,/dev/$vmdisk4 \
-s 4,passthru,0/2/0 \
-s 7,virtio-net,tap4 \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5904,w=1500,h=950 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm4 < /dev/null & sleep 2 && vncviewer 0:4

this is the error I get :

bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory. GPU passthrough might not work properly.bhyve: gvt_d_setup_opregion: Unable to get OpRegion base and length ;bhyve: gvt_d_init: Unable to setup OpRegion ; device emulation initialization error: Operation not supported by device

How can I fix this bug ? A bhyve developer replied to me in this way :

I thought in general we couldn't pass the GPU if it was being used by the host? (Thus you would need to have two GPUs, one to allow the host to display its output and one to passthrough. Unless the host is completely headless).

At this point I think that the intel GPU is actually in use,but I don't know what could keep it in use. Infact :

I've removed the xf86-video-intel driver from the system with : pkg remove xf86-video-intel

2. I haven't any i915kms or legacy intel module inside the memory :

1. kldstat | grep i915 : nothing
2. kldstat | grep kms : nothing
3. kldstat | grep intel : nothing

3. this is my xorg.conf (where is there is no trace of intel driver) : https://pastebin.ubuntu.com/p/hMJpFfmxdN/

I've just booted a linux vm passing thru the RTX 2080 ti using the same parameters used before and it worked. I've detached the HDMI cable from the 1060 and I've attached it to the 2080 ti and I've got what I wanted from the beginning : to choose between virtualize an OS with bhyve or more easily,only using the second monitor without bhyve. Now I want also pass thru the intel GPU.

What video card do I think provides the image I'm looking at on my monitor ? The following scheme is valid to answer this question :

1. monitor AOC + monitor Samsung -> 1 HDMI cable (for the samsung) + 1 DP cable (for the AOC) -> geforce 1060 - without a bhyve-Linux VM

2. monitor AOC + monitor Samsung -> 1 DP cable attached to the AOC and 1 hdmi cable attached to the RTX 2080 ti - if I want to use a bhyve / Linux vm passing the RTX 2080 ti GPU

3. monitor AOC + monitor Samsung -> 1 DP cable attached to the AOC and 1 HDMI cable attached to the Intel GPU - if I want to use a bhyve / Linux VM passing the Intel GPU

The Intel GPU is always kept free as well as the RTX 2080 ti. Even in the BIOS,since I've chosen the 1060 as default GPU,but the Intel GPU is enabled.

[corvink]

In D26209#817317, @marietto2008_gmail.com wrote:

Hello.

I'm running FreeBSD 13.1-RELEASE and I'm trying to passthru my CoffeeLake-S GT2 [UHD Graphics 630] on a Linux VM but it failed.

Seems like your booting your Intel GPU in Legacy Mode. Go into your BIOS and disable CSM.

[marietto2008_gmail.com]

In D26209#818972, @c.koehne_beckhoff.com wrote:

In D26209#817317, @marietto2008_gmail.com wrote:

Hello.

I'm running FreeBSD 13.1-RELEASE and I'm trying to passthru my CoffeeLake-S GT2 [UHD Graphics 630] on a Linux VM but it failed.

Seems like your booting your Intel GPU in Legacy Mode. Go into your BIOS and disable CSM.

Nope. It's not caused by the CSM legacy mode enabled on my BIOS,because I keep it always disabled.

[corvink]

• rebase and split into smaller commits

[corvink]

• don't emulate writes of ASLS. It's not required.

[corvink]

@jhb @markj Any comments?