DMAR: handle affinity for in-memory data structures

[kib]

I am unable to figure out how to make bus_get_domain() to report domain for dmar. dmarX is below acpi0:

# devinfo -p dmar0
dmar0 acpi0 nexus0

so the implementation seems to ask acpi0, which has no idea about DMAR/RHSAS (sub)table.

[jhb]

Yes, the acpi_get_domain() would have to recognize dmar devices and do special handling. Today it just handles _PXM methods on Device objects, and if a device doesn't have a _PXM method it just punts. Presumably the DMAR devices are not Device nodes in the ACPI namespace and thus don't have a _PXM method?

[kib]

Dmars are described by ACPI DMAR table, which enumerates remappers and also provides information about their affinity. There is no ACPI device, and no _PXM.

The dmar driver has the identify() method to instantiate devices as directed by DMAR. Might be some trick is possible there?

[jhb]

So the acpi bus will need some way to recognize special DMAR devices (e.g. a custom ivar) in acpi_get_domain(), and then for DMAR devices it needs to call some other helper routine instead of acpi_parse_pxm() that looks up the domain.

[markj]

I booted the patch on a NUMA intel system I have handy, and it seems to work, though dmar0 belongs to domain 1. Is there a good way to determine which devices use which DMAR instance?

dmar0: <DMA remap> iomem 0xfbffc000-0xfbffcfff on acpi0
dmar0: dmar_attach:390 domain 1
dmar1: <DMA remap> iomem 0xf3ffc000-0xf3ffcfff on acpi0
dmar1: dmar_attach:390 domain 0

sys/x86/iommu/intel_idpgtbl.c

236

692

Should this assignment be done only once? i.e., is this object shared among multiple DMAR instances?

[kib]

In D47254#1077580, @markj wrote:

I booted the patch on a NUMA intel system I have handy, and it seems to work, though dmar0 belongs to domain 1. Is there a good way to determine which devices use which DMAR instance?

dmar0: <DMA remap> iomem 0xfbffc000-0xfbffcfff on acpi0
dmar0: dmar_attach:390 domain 1
dmar1: <DMA remap> iomem 0xf3ffc000-0xf3ffcfff on acpi0
dmar1: dmar_attach:390 domain 0

The dev.<name>.<unit>.%iommu sysctl reports the IOMMU info for corresponding device. In particular, the iommu unit, and the PCIe rid, as calculated from BIOS tables:

dev.igb.0.%iommu: unit=amdiommu0 rid=0x400

sys/x86/iommu/intel_idpgtbl.c

692

This assignment is done only once. Could you please explain why do you think it might happen more?

The object is a container for page table pages for non-id mapped domain, and is used by single domain for single DMAR.

[kib]

In D47254#1077565, @jhb wrote:

So the acpi bus will need some way to recognize special DMAR devices (e.g. a custom ivar) in acpi_get_domain(), and then for DMAR devices it needs to call some other helper routine instead of acpi_parse_pxm() that looks up the domain.

I do not want to push a knowledge about DMAR into acpi. Would something like this work: add a 'fake' bus dmarbus0 below acpi0, with the only non-default method BUS_GET_DOMAIN(), and make all dmarX children of dmarbus0, so the hierarchy would be

dmarX dmarbus0 acpi0 nexus0

Then dmarbus0 method is called and knows how to calculate the domain?

[markj]

In D47254#1077674, @kib wrote:

In D47254#1077580, @markj wrote:

I booted the patch on a NUMA intel system I have handy, and it seems to work, though dmar0 belongs to domain 1. Is there a good way to determine which devices use which DMAR instance?

dmar0: <DMA remap> iomem 0xfbffc000-0xfbffcfff on acpi0
dmar0: dmar_attach:390 domain 1
dmar1: <DMA remap> iomem 0xf3ffc000-0xf3ffcfff on acpi0
dmar1: dmar_attach:390 domain 0

The dev.<name>.<unit>.%iommu sysctl reports the IOMMU info for corresponding device. In particular, the iommu unit, and the PCIe rid, as calculated from BIOS tables:

dev.igb.0.%iommu: unit=amdiommu0 rid=0x400

Thanks. The results look correct to me.

sys/x86/iommu/intel_idpgtbl.c

692

I see, I confused this function with iommu_pgalloc() when I was reading the code.

But, shouldn't this assignment formally be done under DMAR_DOMAIN_PGLOCK?

[kib]

sys/x86/iommu/intel_idpgtbl.c

692

The object is not yet visible for any other thread, until the domain is fully constructed. This is common pattern (issue?) for vm object initialization.

[jhb]

In D47254#1077677, @kib wrote:

In D47254#1077565, @jhb wrote:

So the acpi bus will need some way to recognize special DMAR devices (e.g. a custom ivar) in acpi_get_domain(), and then for DMAR devices it needs to call some other helper routine instead of acpi_parse_pxm() that looks up the domain.

I do not want to push a knowledge about DMAR into acpi. Would something like this work: add a 'fake' bus dmarbus0 below acpi0, with the only non-default method BUS_GET_DOMAIN(), and make all dmarX children of dmarbus0, so the hierarchy would be

dmarX dmarbus0 acpi0 nexus0

Then dmarbus0 method is called and knows how to calculate the domain?

That could work. If you know the domain when the device is created, we could perhaps do something simpler however and add a 'domain' IVAR to ACPI devices and let the identify routine for dmar set that IVAR, and fix acpi_get_domain() (and acpi_get_cpus()) to honor that IVAR if it is set and only fall back to using _PXM if the IVAR isn't set. (We could
also have device enumeration parse _PXM and set the IVAR as well if we go that route.)