arm64: Handle translation faults for thread structures

[markj]

Summary

The break-before-make requirement poses a problem when promoting or
demoting mappings containing thread structures: a CPU may raise a
translation fault while accessing curthread, and data_abort() accesses
the thread again before pmap_fault() can translate the address and
return.

Normally this isn't a problem because we have a hack to ensure that
slabs used by the thread zone are always accessed via the direct map,
where promotions and demotions are rare. However, this hack doesn't
work properly with UMA_MD_SMALL_ALLOC disabled, as is the case with
KASAN configured (since our KASAN implementation does not shadow the
direct map and so tries to force the use of the kernel map wherever
possible).

Fix the problem by modifying data_abort() to handle translation faults
in the kernel map without dereferencing "td", i.e., curthread, and
without enabling interrupts. pmap_fault() has special handling for
translation faults which makes it safe to call. Then, revert the
aforementioned hack.

[andrew]

sys/arm64/arm64/trap.c

293–294

Is there a reason to not call pmap_fault unconditionally here? We could then either add a flag to skip the call later or move it up to the relavant condition blocks.

[markj]

sys/arm64/arm64/trap.c

293–294

Yes: pmap_fault() may acquire the pmap lock to handle other types of faults, which can't be done while interrupts are disabled. Interrupts have to remain disabled here since handlers might try to dereference curthread.

We could skip a second pmap_fault() call here, but if it fails the first time then won't this likely be a fatal data abort anyway?

[andrew]

sys/arm64/arm64/trap.c

293–294

In that case I think we just use pmap_klookup directly here as all we need to know is if there is a mapping for the virtual address, even if it's currently in a break-before-make sequence.

[markj]

Use pmap_klookup() directly

[alc]

sys/arm64/arm64/trap.c

293–294

We could skip a second pmap_fault() call here, but if it fails the first time then won't this likely be a fatal data abort anyway?

I would suggest adding a sentence relating to this, e.g., "In principle, if pmap_klookup() fails, there is no need to call pmap_fault() below, but avoiding that call is not worth the effort."