Paths

Table of Contentst

VM detection: Add QEMU; Add prefix match; Change prioritization
Needs ReviewPublic
Actions

Authored by olce on Thu, Feb 20, 5:21 PM.

Details

Reviewers

stevek
jhb
imp
kib

Summary

Introduce VM_GUEST_QEMU. As VM_GUEST_BHYVE, VM_GUEST_PARALLELS and
VM_GUEST_NVMM, it has no specific code associated with it, but is
declared just so that "qemu" can be reported by sysctl 'kern.vm_guest'.

Add prefix match, and change matching of Parallels in
'smbios.system.product' to be prefix-based, as the current exact name:

Does not match Parallels running on Apple silicon (arm64).
Probably does not match Parallels running on amd64 either, as the string seem more similar to what is contained in 'smbios.planar.product' ("Virtual Platform", and not "Virtual Machine), which we don't use.

According to an analysis of its code, the only trustable indication of
QEMU running is "QEMU" in 'smbios.system.maker', as other strings may be
overriden by machine descriptions. However, since QEMU enables
overriding any of the SMBIOS strings, we keep the match on
'smbios.system.product' to increase chances of detection, and change it
as a prefix match given QEMU may also insert machine details between
"QEMU" and "Virtual Machine".

Matches of the loader-filled SMBIOS strings are now performed as follows:

'smbios.system.maker' (System's Manufacturer, Type 1)
'smbios.system.product' (System's Product Name, Type 1)
'smbios.bios.vendor' (Platform Firmware's Vendor Name, Type 0)

Before this change, 1 was not matched, and 2 and 3 were swapped as we
believe that case 2 has more chances to be discriminating than case 3.
Actually, a "continue" mechanism would allow proceeding to case 2 if
case 3 only determined VM_GUEST_VM (i.e., running in some unspecified
VM) in the hope that case 2 would refine it. It has been removed given
the swap.

Test Plan

Ask people to boot FreeBSD under several virtual machines on amd64 and arm64.

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Build Status

Buildable 62549
Build 59433: arc lint + arc unit

Event Timeline

olce requested review of this revision.Thu, Feb 20, 5:21 PM

olce created this revision.

Harbormaster completed remote builds in B62549: Diff 151256.Thu, Feb 20, 5:21 PM

So there are multiple ways to run under qemu: system emulation, kvm, nvmm, and hvf. These overlap with others in the list (parallels uses hvf also).

Each of these environments has its own quirks. But they seem to all be conflated...

In D49084#1119093, @imp wrote:

So there are multiple ways to run under qemu: system emulation, kvm, nvmm, and hvf. These overlap with others in the list (parallels uses hvf also).

Each of these environments has its own quirks. But they seem to all be conflated...

May be, yeah. Unless I'm mistaken, we don't "support" hvf (but maybe it doesn't need quirks; we don't have quirks for Parallels).

I'm personally far from being a specialist in this area, so trying to gather as much input as possible. Also, I'm not trying to fix everything but at least what I could given the recent mailing list input and threads ("[Retitled!] some under-VM detections for non-amd64 may be broken" and "Warning: FreeBSD using its (in partition) SWAP space when running under Parallels on aarch64 macOS (M4 MAX)") and a review of the existing code, without disturbing other cases. I searched through KVM's code as well as QEMU's to see which kind of strings are produced (I might have missed some things).

In the end, maybe having a single vm_guest value is too narrow, but I'm not trying to change that here (at first).

In D49084#1119093, @imp wrote:

So there are multiple ways to run under qemu: system emulation, kvm, nvmm, and hvf. These overlap with others in the list (parallels uses hvf also).

Each of these environments has its own quirks. But they seem to all be conflated...

On x86, if QEMU uses KVM, we will report kvm instead of qemu in kern.vm_guest, because we rely on CPUID info first (before the SMBIOS info passed by the loader, and modified here). The latter is not the case on arm64. Putting the test on smbios.system.maker first, with QEMU being part of the list, means that on arm64 we will favor QEMU over KVM (provided that KVM works on ARM, which I don't even know). That could be fixed by changing the order, although I agree this may complicate next evolutions (maybe a kind of priority mechanism is in order, or multiple flags linked to the quirks applied, or...).

Probably does not match Parallels running on amd64 either, as the string seem more similar to what is contained in 'smbios.planar.product' ("Virtual Platform", and not "Virtual Machine), which we don't use.

I later tested Parallels on a macMini 2018 (so: amd64) with a amd64 FreeBSD boot media and the detection worked fine. Parallels just changed the text's convention to be more explicit when the new platform was added --and only did so on the new platform.

UTM on macOS can use either QEMU or Apple's VM as I understand. But I do not know of FreeBSD is actually operable with Apple's VM in use. If operable, Apple's VM likely leads to another new name to test for? (This might be another example of: Treat FreeBSD as-if it was a Linux variant, like I did for Parallels to get it basically working there. I then disabled virtio_gpu via /boot/device.hints so that efifb would be used, which worked fine for my purposes.)

In D49084#1119093, @imp wrote:

So there are multiple ways to run under qemu: system emulation, kvm, nvmm, and hvf. These overlap with others in the list (parallels uses hvf also).

For macOS, likely the qemu is typically the forked variant that is in UTM, just to add to the source of variations. I'm not sure of anyone has tested FreeBSD via qemu without using UTM for macOS.

(I do not use UTM because I want directly boot media that I also use to boot the Windows Dev Kit 2023 and the RPi5 that I have access to. UTM does not support such on macOS but Parallels does.)

In D49084#1119094, @olce wrote:

In D49084#1119093, @imp wrote:

So there are multiple ways to run under qemu: system emulation, kvm, nvmm, and hvf. These overlap with others in the list (parallels uses hvf also).

Each of these environments has its own quirks. But they seem to all be conflated...

May be, yeah. Unless I'm mistaken, we don't "support" hvf (but maybe it doesn't need quirks; we don't have quirks for Parallels).

I use qemu + hvf all the time. It just works. No quirks are needed, but I worry that you'll start doing quirks for qemu-over-something-else and that will mess up my working setup.

Parallels 100% uses hvf under the hood. It's designed so guests don't really need to know.

I'm personally far from being a specialist in this area, so trying to gather as much input as possible. Also, I'm not trying to fix everything but at least what I could given the recent mailing list input and threads ("[Retitled!] some under-VM detections for non-amd64 may be broken" and "Warning: FreeBSD using its (in partition) SWAP space when running under Parallels on aarch64 macOS (M4 MAX)") and a review of the existing code, without disturbing other cases. I searched through KVM's code as well as QEMU's to see which kind of strings are produced (I might have missed some things).

Yea, QEMU just reports its in the loop somehow. It doesn't communicate, via smbios, how it's involved.

Existing code, in FreeBSD, can be a bit problematic. Some of the detection is good, some is a quick hack that might not be right. The paper-trail on it was a bit thin last time I had to delve into the area.

In the end, maybe having a single vm_guest value is too narrow, but I'm not trying to change that here (at first).

It used to matter a lot so we'd get hz right, but that doesn't matter anymore (though we bogusly still do it sometimes). Hz hasn't mattered since mav@ made the kernel tickless.

But I'd think it would matter a lot more if we're under KVM or HVF than the agent that got us there for the quirks. But if we're not doing quirks, it's all cosmetic so what's it matter?

In D49084#1119144, @markmi_dsl-only.net wrote:

UTM on macOS can use either QEMU or Apple's VM as I understand. But I do not know of FreeBSD is actually operable with Apple's VM in use. If operable, Apple's VM likely leads to another new name to test for? (This might be another example of: Treat FreeBSD as-if it was a Linux variant, like I did for Parallels to get it basically working there. I then disabled virtio_gpu via /boot/device.hints so that efifb would be used, which worked fine for my purposes.)

I was under the impression Apple's VM in this case was parallel's, and UTM just provided a fancy / different GUI for it.

I was under the impression Apple's VM in this case was parallel's, and UTM just provided a fancy / different GUI for it.

I doubt that if one has not bought or installed Parallels, that UTM using Apple's VM will report anything indicating Parallels: bad branding, even if the internals really are still largely the same.

In D49084#1119169, @markmi_dsl-only.net wrote:

I was under the impression Apple's VM in this case was parallel's, and UTM just provided a fancy / different GUI for it.

I doubt that if one has not bought or installed Parallels, that UTM using Apple's VM will report anything indicating Parallels: bad branding, even if the internals really are still largely the same.

As far as FreeBSD is concerned, that's the same. We don't care about third party branding

It used to matter a lot so we'd get hz right, but that doesn't matter anymore (though we bogusly still do it sometimes). Hz hasn't mattered since mav@ made the kernel tickless.

Quoting https://docs.freebsd.org/en/books/handbook/virtualization/ in its "24.2.2. Configuring FreeBSD on Parallels":

Set Boot Loader Variables

The most important step is to reduce the kern.hz tunable to reduce the CPU utilization of FreeBSD under the Parallels environment. This is accomplished by adding the following line to /boot/loader.conf:

kern.hz=100
Without this setting, an idle FreeBSD Parallels guest will use roughly 15% of the CPU of a single processor iMac®. After this change the usage will be closer to 5%.

If installing FreeBSD 14.0 or later, and CPU utilization is still high, add the following additional line to /boot/loader.conf:

debug.acpi.disabled="ged"

(No mention is made of the automatic kern.hz assignment for at least amd64/i386.)

"24.3.2. Configuring FreeBSD on VMware Fusion" has the same sort of text but I've not checked if it is automatic in that type of context.

I've not tested for the distinction in overhead in any context. Checking kern.eventtimer.periodic indicates 0 . (I assume that means that tickless is not required but is the default, in which case an odd set up might want the kern.hz assignment because of using tick-based.)

Note: I did get an Email where someone reported that they quit using kern.hz=100 because it lead to time skew problems. Another thing that I've not tested for.

In D49084#1119158, @imp wrote:

I use qemu + hvf all the time. It just works. No quirks are needed, but I worry that you'll start doing quirks for qemu-over-something-else and that will mess up my working setup.

I do not plan to add specific quirks for QEMU.

It used to matter a lot so we'd get hz right, but that doesn't matter anymore (though we bogusly still do it sometimes). Hz hasn't mattered since mav@ made the kernel tickless.
But I'd think it would matter a lot more if we're under KVM or HVF than the agent that got us there for the quirks. But if we're not doing quirks, it's all cosmetic so what's it matter?

Automatic hz tuning has been the initial motivation for this change. The kernel is tickless (for timers and events) but hz is still used a lot, in particular when waiting for (normally infrequent) resource contention or wait events. From a cursory look, it indeed doesn't seem to matter much, but I'm not sure that there are no cases where this could really matter. Do we have more info/pointers?

For reference, taken from my list report for aarch64 Parallels on macOS. I'll likely separately submit one from amd64 Parallels on macOS as well.

# kenv | grep -F smbios
hint.smbios.0.mem="0x1a3f820000"
smbios.bios.reldate="Tue, 04 Feb 2025 17:47:55"
smbios.bios.revision="0.1"
smbios.bios.vendor="Parallels International GmbH."
smbios.bios.version="20.2.1 (55876)"
smbios.chassis.maker="Parallels International GmbH."
smbios.chassis.serial=" "
smbios.chassis.tag=" "
smbios.chassis.type="Unknown"
smbios.chassis.version=" "
smbios.memory.enabled="117440512"
smbios.planar.location="None"
smbios.planar.maker="Parallels ARM Virtual Machine"
smbios.planar.product="Parallels ARM Virtual Platform"
smbios.planar.serial="None"
smbios.planar.tag="None"
smbios.planar.version="0.1"
smbios.socket.enabled="1"
smbios.socket.populated="1"
smbios.system.family="Parallels VM"
smbios.system.maker="Parallels International GmbH."
smbios.system.product="Parallels ARM Virtual Machine"
smbios.system.serial= *** OMITTED ***
smbios.system.sku="Parallels_ARM_VM"
smbios.system.uuid= *** OMITTED ***
smbios.system.version="0.1"
smbios.version="3.0"

In D49084#1119184, @olce wrote:

In D49084#1119158, @imp wrote:

I use qemu + hvf all the time. It just works. No quirks are needed, but I worry that you'll start doing quirks for qemu-over-something-else and that will mess up my working setup.

I do not plan to add specific quirks for QEMU.

It used to matter a lot so we'd get hz right, but that doesn't matter anymore (though we bogusly still do it sometimes). Hz hasn't mattered since mav@ made the kernel tickless.
But I'd think it would matter a lot more if we're under KVM or HVF than the agent that got us there for the quirks. But if we're not doing quirks, it's all cosmetic so what's it matter?

Automatic hz tuning has been the initial motivation for this change. The kernel is tickless (for timers and events) but hz is still used a lot, in particular when waiting for (normally infrequent) resource contention or wait events. From a cursory look, it indeed doesn't seem to matter much, but I'm not sure that there are no cases where this could really matter. Do we have more info/pointers?

There have been a dozen or so threads. Hz just doesn't matter. The performance overhead is long gone and almost nobody bothers to tune it anymore. It's the default for things like bhyve-vm. It's not been important in 5 or 10 years. The handbook is just out of date since our docs team in that same period has been inactive.

Here is from amd64 Parallels on amd64 macOS:

# kenv | grep -F smbios
hint.smbios.0.mem="0xae9ec000"
smbios.bios.reldate="02/04/2025"
smbios.bios.revision="20.2"
smbios.bios.vendor="Parallels International GmbH."
smbios.bios.version="20.2.1 (55876)"
smbios.chassis.maker="Parallels International GmbH."
smbios.chassis.serial=" "
smbios.chassis.tag=" "
smbios.chassis.type="Unknown"
smbios.chassis.version=" "
smbios.memory.enabled="8388608"
smbios.planar.maker="Parallels International GmbH."
smbios.planar.product="Parallels Virtual Platform"
smbios.planar.serial="None"
smbios.planar.version="None"
smbios.socket.enabled="1"
smbios.socket.populated="1"
smbios.system.family="Parallels VM"
smbios.system.maker="Parallels International GmbH."
smbios.system.product="Parallels Virtual Platform"
smbios.system.serial= *** OMITTED ***
smbios.system.sku="Undefined"
smbios.system.uuid= *** OMITTED ***
smbios.system.version="None"
smbios.version="2.7"

Ignoring kern.hz : Is there any other reason that it is important for FreeBSD to have the explicit indication that it is running under a VM?

(My "bulk -ca" experiment turned out to be without FreeBSD having the indication from this code.)

In D49084#1119188, @markmi_dsl-only.net wrote:

Ignoring kern.hz : Is there any other reason that it is important for FreeBSD to have the explicit indication that it is running under a VM?

(My "bulk -ca" experiment turned out to be without FreeBSD having the indication from this code.)

virtualbox guest tools, I think, may exit doing nothing if not running under virtual box. HyperV has special code activated. I think xen does too. I don't think those two have useland tools depending on it

virtualbox guest tools, I think, may exit doing nothing if not running under virtual box. HyperV has special code activated. I think xen does too. I don't think those two have useland tools depending on it

So it sounds like my experimental aarch64 FreeBSD activity under Parallels on aarch64 macOS was likely good/normal/as-expected instead of messed up by the lack of identification as a VM context. (I was not using any guest tools and I told Parallels that FreeBSD was a "Other Linux". That seems to have done a ACPI based boot, rather than a direct linux-kernel boot.)

May be the VM status should only be noted for the examples with such a known-that-FreeBSD-needs-to-know-its-VM-type status? And the rest should just use VM_GUEST_NO? I.e.:

# sysctl kern.vm_guest
kern.vm_guest: none

# sysctl kern.hz
kern.hz: 1000
}

would result unless it was known that something special was needed? That might cut down on the tracking/updates needed, at least for main and later.

(The kenv smbios* information likely should still be present.)

In D49084#1119188, @markmi_dsl-only.net wrote:

Ignoring kern.hz : Is there any other reason that it is important for FreeBSD to have the explicit indication that it is running under a VM?

Determining the specific "VM" FreeBSD is running on currently matters:

For probing early TSC frequency via hypercalls (VM_GUEST_XEN, VM_GUEST_VMWARE).
To enable probing for Hyper-V specific buses, I/O APIC assigning on amd64 and GICv3 attach on arm64 (VM_GUEST_HV).
To enable GuestRPC code and to possibly disable x2apic (VM_GUEST_VMWARE).
To know whether to request KVM features from CPUID (VM_GUEST_KVM).
To degrade TSC quality as a timecounter (VM_GUEST_VBOX).

The only common behavior to all VMs (i.e., if vm_guest is not VM_GUEST_NO) is to set hz to HZ_VM (100).

As said in the description, VM_GUEST_BHYVE, VM_GUEST_PARALLELS and VM_GUEST_NVMM are "cosmetic" in the sense that they have no specific behaviors/quirks attached.

In D49084#1119186, @imp wrote:

In D49084#1119184, @olce wrote:

Automatic hz tuning has been the initial motivation for this change. (snip)

(snip) Hz just doesn't matter. (snip)

Well, I've read a bit on the topic, and apparently hz should not matter except for timekeeping. Experimenting with idle VirtualBox VMs, I see a minimum difference of 0,20% CPU (sometimes going as high as 0,60% CPU) on Zen2 cores in CPU consumption, that's indeed tiny, but not zero. If you (or someone else) wants to remove the automatic tuning to HZ_VM, please go ahead.

In any case, beyond the hz topic, there's also that people will inevitably ask about kern.vm_guest not set whereas running in a virtual machine, and we have already set a precedent with VM_GUEST_BHYVE, VM_GUEST_PARALLELS and VM_GUEST_NVMM (which are only cosmetic but for hz). So I'd rather for now make sure that kern.vm_guest is not none under QEMU, even if in the end this is essentially cosmetic. Having a special value seems slightly better, but VM_GUEST_VM (generic) could also work.

Given what I wrote about the effect of VM_GUEST_KVM above, it appears that it doesn't really matter if, on arm64, QEMU + KVM is detected as QEMU (or generic) instead of KVM (as on amd64, which is important there because of the CPUID gating). None of your QEMU use cases would be disturbed at all. So I'd rather just commit the current proposal as it is, adding info from this Phab differential into the commit message/comments.

In any case, beyond the hz topic, there's also that people will inevitably ask about kern.vm_guest not set whereas running in a virtual machine, and we have already set a precedent with VM_GUEST_BHYVE, VM_GUEST_PARALLELS and VM_GUEST_NVMM (which are only cosmetic but for hz). So I'd rather for now make sure that kern.vm_guest is not none under QEMU, even if in the end this is essentially cosmetic. Having a special value seems slightly better, but VM_GUEST_VM (generic) could also work.

The other side of this is the potential need to bother with the likes of identifying, say, the Apple VM. Otherwise it would likely have the problem of kern.vm_guest being none (if FreeBSD can run under it). The same for any other type of VM not explicitly covered. May be the documentation also needs a note about how any "unknown VMs" are handled and, so, to expect none in such contexts. Then the status for the 3 VM_GUEST_ above would be optional for which way to handle them. Making it clear in the documentation that they are cosmetic might be a good idea if they are left in place as well: none is more explicit about there being nothing special going on.

In D49084#1119353, @olce wrote:

The only common behavior to all VMs (i.e., if vm_guest is not VM_GUEST_NO) is to set hz to HZ_VM (100).

Forgot to mention that are a number of other tweaks performed when vm_guest is not VM_GUEST_NO, i.e., for any detected VM:

Disabling CLFLUSH under virtualization environments (initially a workaround for some Xen issue).
Using the first 64kb of physical memory on amd64 (not fearing they are clobbered by the BIOS on suspend/resume).
Applying some processor errata.
Waving TSC skew computation in DTrace.
Setting allowed IRQs to 0 when attaching ACPI HPET (workaround introduced for QEMU and VirtualBox).
Avoid writing to some debug MSRs in certain circumstances (in hwpmc).
Avoid blacklisting some PCI chipsets.
Don't ignore the ACPI FADT NO_VGA flag.
Have vt's (x86-only) VGA boot in text mode by default.
Elide printing "calcru: runtime went backwards".
Stop disabling x2APIC on certain CPU models.
Stop applying the Zenbleed chicken bit.
Don't try to re-enable AMD topology extension.
Calibrate LAPIC from CPUID info considered trustable.
Same for TSC.
Don't necessarily consider TSC invariant for "recent" Intel/AMD processors.

(17. Set hz to 100, unless manually overridden.)

Some might need revision, but it seems at first a good idea to apply them all pro-actively in order to avoid potential problems (pushing into the direction of detecting VMMs).

In D49084#1119362, @markmi_dsl-only.net wrote:

The other side of this is the potential need to bother with the likes of identifying, say, the Apple VM. Otherwise it would likely have the problem of kern.vm_guest being none (if FreeBSD can run under it). The same for any other type of VM not explicitly covered.

As long as we have global tunings for running under a VM, this is unescapable. Unrecognized VMMs may work well without those tweaks though, but this should probably be validated before "tweaks" are removed, not the other way around, meaning that, by default, we should strive to detect them.

Then the status for the 3 VM_GUEST_ above would be optional for which way to handle them.

Yes, they could be replaced with VM_GUEST_VM.

Making it clear in the documentation that they are cosmetic might be a good idea if they are left in place as well: none is more explicit about there being nothing special going on.

none is saying "nothing is going on" and corresponds to VM_GUEST_NO. Those VM_GUEST_* do not do that, they actually enable all common tweaks (which, as we have just seen, are much more numerous than just the hz tweak). If substituting these with VM_GUEST_VM, then generic is the expected output. none really means no VMM has been detected (and no tweaks are applied at all). They are different.

olce added a subscriber: markj.Fri, Feb 21, 5:12 PM

As requested.

UTM on Apple M3 (ARM) running aarch64 FreeBSD VM (using native hypervisor):

olivier@vm:~ $ kenv | grep -F smbios
hint.smbios.0.mem="0x13f800000"
smbios.bios.reldate="02/06/2015"
smbios.bios.revision="0.0"
smbios.bios.vendor="EFI Development Kit II / OVMF"
smbios.bios.version="0.0.0"
smbios.chassis.maker="QEMU"
smbios.chassis.type="Other"
smbios.chassis.version="virt-9.1"
smbios.memory.enabled="4194304"
smbios.socket.enabled="1"
smbios.socket.populated="1"
smbios.system.maker="QEMU"
smbios.system.product="QEMU Virtual Machine"
smbios.system.uuid="68f2f8ab-30aa-428a-8b40-db0040de9b43"
smbios.system.version="virt-9.1"
smbios.version="3.0"

UTM on Intel Macbook running FreeBSD x86 VM (so using native hypervisor):

olivier@vm:~ $ kenv | grep -F smbios
hint.smbios.0.mem="0x7f942000"
smbios.bios.reldate="02/06/2015"
smbios.bios.revision="0.0"
smbios.bios.vendor="EFI Development Kit II / OVMF"
smbios.bios.version="0.0.0"
smbios.chassis.maker="QEMU"
smbios.chassis.type="Other"
smbios.chassis.version="pc-q35-9.1"
smbios.memory.enabled="4194304"
smbios.socket.enabled="1"
smbios.socket.populated="1"
smbios.system.maker="QEMU"
smbios.system.product="Standard PC (Q35 + ICH9, 2009)"
smbios.system.uuid="eddddc62-d0e8-4c63-961c-6cafaf7f4b0a"
smbios.system.version="pc-q35-9.1"
smbios.version="2.8"

UTM on Intel Macbook running FreeBSD ARM VM (so emulated):

olivier@ARMvm:~ $ kenv | grep -F smbios
hint.smbios.0.mem="0x13f800000"
smbios.bios.reldate="02/06/2015"
smbios.bios.revision="0.0"
smbios.bios.vendor="EFI Development Kit II / OVMF"
smbios.bios.version="0.0.0"
smbios.chassis.maker="QEMU"
smbios.chassis.type="Other"
smbios.chassis.version="virt-9.1"
smbios.memory.enabled="4194304"
smbios.socket.enabled="1"
smbios.socket.populated="1"
smbios.system.maker="QEMU"
smbios.system.product="QEMU Virtual Machine"
smbios.system.uuid="f985d4e7-2c95-4ade-9442-68f91c986f31"
smbios.system.version="virt-9.1"
smbios.version="3.0

In D49084#1119428, @olivier wrote:

As requested.

Thanks! This confirms my reading of QEMU's code and that using smbios.system.maker to detect QEMU is probably the best solution (others still can be applied in addition to it). And the datapoints are in any case useful should we later want to change how the detection works.