Add per-arch flags to struct vm_map_entry

[andrew]

Summary

And pass it to pmap functions that create or modify page table entries
so they can use it to set page table fields that don't fit into the
memory protection attributes.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation

[andrew]

I've only added the arm64 pmap changes for now. Will add all architectures if there are no problems with this approach.

[kib]

When I added PKRU support for amd64, I did not extended vm_map. Instead, I added off-vm_map rangeset (see sys/rangeset.h and kern/subr_rangeset.c) to pmap.

IMO it is more correct WRT layering and does not incur even a small cost on architectures that do not use the new data element.

[andrew]

I've looked at moving it to a rangeset, the main issue with that is we don't know when userspace unmaps protected memory in pmap.c. We can't rely on pmap_remove as it can be called in swapout cases.

Another option would be to reserve VM_PROT_* bits for architecture specific use. We have 3 spare bits there so could add VM_PROT_ARCH, however I'm not sure how that would interact with VM_PROT_ALL.

[kib]

In D39449#900308, @andrew wrote:

I've looked at moving it to a rangeset, the main issue with that is we don't know when userspace unmaps protected memory in pmap.c. We can't rely on pmap_remove as it can be called in swapout cases.

If this is the main obstacle, we can add e.g. pmap_unmap() method and call it on vm_map_remove(), or even we can enhance pmap_remove() by informing what specifically is the reason for the call.

Another option would be to reserve VM_PROT_* bits for architecture specific use. We have 3 spare bits there so could add VM_PROT_ARCH, however I'm not sure how that would interact with VM_PROT_ALL.

[jrtc27]

In D39449#900308, @andrew wrote:

I've looked at moving it to a rangeset, the main issue with that is we don't know when userspace unmaps protected memory in pmap.c. We can't rely on pmap_remove as it can be called in swapout cases.

Another option would be to reserve VM_PROT_* bits for architecture specific use. We have 3 spare bits there so could add VM_PROT_ARCH, however I'm not sure how that would interact with VM_PROT_ALL.

CheriBSD is using two of those; whilst FreeBSD is technically free to use them, it would be a real pain for us downstream I'm sure, so if that can be avoided it would be best.

[markj]

In D39449#912032, @jrtc27 wrote:

In D39449#900308, @andrew wrote:

I've looked at moving it to a rangeset, the main issue with that is we don't know when userspace unmaps protected memory in pmap.c. We can't rely on pmap_remove as it can be called in swapout cases.

Another option would be to reserve VM_PROT_* bits for architecture specific use. We have 3 spare bits there so could add VM_PROT_ARCH, however I'm not sure how that would interact with VM_PROT_ALL.

CheriBSD is using two of those; whilst FreeBSD is technically free to use them, it would be a real pain for us downstream I'm sure, so if that can be avoided it would be best.

The original issue, that the pmap layer doesn't know when userspace unmaps protected memory, is now handled in principle by having separate pmap_remove() and pmap_map_delete() routines.

[andrew]

Not needed