Map arm64 pci config memory as non-posted

[andrew]

Summary

On arm64 PCI config memory is expected to be mapped with a non-posted
device type. To handle this use the new bus_map_resource support in
arm64 to map memory with the new VM_MEMATTR_DEVICE_NP attribute. This
memory has already been allocated and activated, it just needs to be
mapped.

[manu]

Maybe it would be simpler and more clear to have a new memory type for struct resource for PCI config region that all the bus_* routines will know ?

[andrew]

This seemed to be the consensus when I asked how best to do it on IRC.

[jhb]

Are there SYS_RES_MEMORY regions for which you don't want VM_MEMATTR_DEVICE_NP?

[andrew]

In D30079#678549, @jhb wrote:

Are there SYS_RES_MEMORY regions for which you don't want VM_MEMATTR_DEVICE_NP?

All non-PCI memory should use the faster nGnRE memory, however the ordering isn't strong enough for PCI config space. I would like to change VM_MEMATTR_DEVICE to use nGnRE mappings, however will need to ensure PCIe still works.

[andrew]

Also on M1 the memory types are backwards so PCI memory will need to be mapped with nGnRE while other MMIO memory will be nGnRnE.

[andrew]

Rebase

[mmel]

There is also a tegra pcie driver that is common to arm and arm64 , so we should define VM_MEMATTR_DEVICE_NP for arm as well. But I'm not sure if we have the rest of the necessary infrastructure implemented on arm.

[andrew]

Map the Tegra PCIe config memory as non-posted

[andrew]

In D30079#777484, @mmel wrote:

There is also a tegra pcie driver that is common to arm and arm64 , so we should define VM_MEMATTR_DEVICE_NP for arm as well. But I'm not sure if we have the rest of the necessary infrastructure implemented on arm.

We use bus_space_map directly in the Tegra driver, so can pass the appropriate flag to it.

[mmel]

Ahh right, thanks.

[kevans]

I had to do this kind of nasty feeling patch to get devices right on M1: https://people.freebsd.org/~kevans/m1/pci-mapping.diff -- I don't know if there's a better way to have done that, or if a custom bus_activate_resource that handles the mapping is really just the way to go.

[andrew]

Does the M1 work if you have this patch, D34333, and the following change?

diff --git a/sys/arm64/include/vm.h b/sys/arm64/include/vm.h
index e479aab52e26..9cdd92b7284e 100644
--- a/sys/arm64/include/vm.h
+++ b/sys/arm64/include/vm.h
@@ -40,7 +40,7 @@
  * VM_MEMATTR_DEVICE can be changed to VM_MEMATTR_DEVICE_nGnRE when
  * the PCI drivers use VM_MEMATTR_DEVICE_NP for their config space.
  */
-#define        VM_MEMATTR_DEVICE               VM_MEMATTR_DEVICE_nGnRnE
+#define        VM_MEMATTR_DEVICE               VM_MEMATTR_DEVICE_nGnRE
 #define        VM_MEMATTR_DEVICE_NP            VM_MEMATTR_DEVICE_nGnRnE

 #ifdef _KERNEL

[kevans]

In D30079#885639, @andrew wrote:

Does the M1 work if you have this patch, D34333, and the following change?

Negative, ish... I note that it triggers an overflow in smmu.c that I just hacked around, but with this I don't even get to COPYRIGHT. I'll dig into why a bit later.