Update patch.

Update patch and summary.

Reworked patch and updated summary.

Address multiple comments:

• enabled UMA_MD_SMALL_ALLOC unconditionally
• removed redundant include guard in ppc uma_machdep
• removed obsolete u_int8_t types

In D45163#1030231, @mjg wrote:

why even support something like this?

instead, if pages can't be all be allocated from one domain, the code could be patched to free up whatever it did and try a different one

Remove redundant m == NULL check.

Simplify vm_thread_stack_back page allocation loop.

Add missing arm vmparam.h updates.

Removed stray UMA code from diff.

Address @markj 's comment - uma_small_alloc code deduplication was carved out into a separate revision.

In D45045#1027214, @kib wrote:

Why not M_NOFREE?

M_NOFREE was already taken by sys/mbuf.h, M_NEVERFREED is one of the alternative names proposed by @alc .

Address @mhorne 's and @kib 's comments:

As promised, I've removed some redundant bits from the patch, it should be a bit clearer now.

Regenerate and simplify patch.

Address @markj 's comments and fix a couple of issues:

• Certain kstack KVA chunks were released back to the parent arena with improper alignment, causing vmem_xfree to panic
• swapping in thread kstacks triggered a panic as the former code relied on vm_page_grab_pages

Sorry for the delay, @markj reported a panic when booting this patch on a NUMA machine and it took me a while to set up a NUMA environment and properly fix the issue.

Update patch to track and properly release kstacks to domain arenas, address @kib 's comments.

In D44565#1016805, @markj wrote:

It's not clear to me why we don't extend the vm_memmap structure instead.

Stepping back for a second, the goal of this patch is not really clear to me. I can see two possibilities:

• We want to create a fake NUMA topology, e.g., to make it easier to use bhyve to test NUMA-specific features in guest kernels.
• We want some way to have bhyve/vmm allocate memory from multiple physical NUMA domains on the host, and pass memory affinity information to the guest. In that case, vmm itself needs to ensure, for example, that the VM object for a given memseg has the correct NUMA allocation policy.

I think this patch ignores the second goal and makes it harder to implement in the future.

You're right, the primary goal was to have a way of faking NUMA topologies in a guest for kernel testing purposes. I did consider the second goal but ultimately decided to focus on the "fake" bits first and implement the rest in a separate patch.
I'll rework the patch so that it covers both goals.

It also appears to assume that each domain can be described with a single PA range, and I don't really understand why vmm needs to know the CPU affinity of each domain.

I'm not that happy about directly specifying PA ranges directly. The only other thing I could think of is to let the user specify the amount of memory per-domain and let bhyve deal with PA ranges, do you think that this is a more sane approach?
As for the CPU affinities, these are needed for SRAT but that can be done purely from userspace. I've kept them in vmm in case we might want to get NUMA topology info using bhyvectl, but I guess that information can be obtained from the guest itself. I'll remove the cpusets.

Address @kib 's comments.

Remove commented lines.

Apologies for the delayed response.
@markj and I will go through the patch once more during next week, so it should get commited soon.

Rebase and address comments by @alc , @kib , and @dougm.

Committed in R9:607c4b857a65b68289e9d7f86a8855def93c09f0.

Address @kib 's comments.

Aside from the issue @markj pointed out, LGTM.

Address @jhb 's comments.

In D42405#976891, @jhb wrote:

If you merge this first, then the breakpoint patch doesn't need 'handled = 0', right?

Address @jhb 's comments.

In D38852#976094, @markj wrote:

In D38852#975886, @bojan.novkovic_fer.hr wrote:

In D38852#957605, @markj wrote:

One other open issue that we didn't discuss much yet is what to do on 32-bit systems. Since the kernel virtual address space is much more limited there, it perhaps (probably?) doesn't make much sense to change the existing scheme. This is especially true if we start trying to align virtual mappings of stacks as Alan suggests.

Given the recent decision to deprecate and potentially remove support for 32-bit systems with 15.0, do you think that this is still an issue we should address?

Yes - the actual removal of 32-bit kernels isn't going to happen anytime soon. 15.0 is two years out, and it's not entirely clear to me that we're ready to remove 32-bit ARM support.

But, can't we handle this simply by making vm_kstack_pindex() use the old KVA<->pindex mapping scheme if _ILP32 is defined?

Address @markj 's comments:

• multiline comment style fixes
• use old mapping scheme on 32-bit systems

In D38852#957605, @markj wrote:

One other open issue that we didn't discuss much yet is what to do on 32-bit systems. Since the kernel virtual address space is much more limited there, it perhaps (probably?) doesn't make much sense to change the existing scheme. This is especially true if we start trying to align virtual mappings of stacks as Alan suggests.

Address @alc 's comments.

Upload diff with full context.

In D41633#969928, @markj wrote:

Would you like to do the honours of reverting the vm_fault change? :)

In D41633#969403, @markj wrote:

Sorry for the delayed reply. The above-mentioned riscv pmap commits are in main now - @bojan.novkovic_fer.hr would it be possible to rebase on top of that? Then I can commit this patch, and we can finally revert D19670.

Rebased the patch.

Address @corvink 's comments.

Address @corvink 's comments.

Address @corvink's comments, unrelated exception reflection change was split into its own patch (D42405), updated commit message to be more clear.

In D40772#966839, @eugen_grosbein.net wrote:

Note this became more important since we have ASLR turned on for 64 bit processes since 13.2-RELEASE. And ASLR adds great deal of fragmentation. It leads to significant performance degradation over long run due to superpages becoming unusable due memory fragmentation.