add UMA_ZONE_NOTRIM & use it for the ktls_buffer zone

[gallatin]

Summary

The ktls buffer zone allocates 16k contiguous buffers, and often needs to call vm_page_reclaim_contig_domain_ext() to free up contiguous memory, which can be expensive. Web servers which have a daily pattern of peaks and troughs end up having UMA trim the ktls_buffer_zone when they are in their trough, and end up re-building it on the way to their peak.

Rather than calling vm_page_reclaim_contig_domain_ext() multiple times on a daily basis, lets mark the ktls_buffer_zone with a new UMA flag, UMA_ZONE_NOTRIM. This disables UMA_RECLAIM_TRIM on the zone, but allows UMA_RECLAIM_DRAIN* operations, so that if we become extremely short of memory (vm_page_count_severe()), the uma reclaim worker can still free up memory.

Note that UMA_ZONE_UNMANAGED already exists, but can never be drained or trimmed, so it may hold on to memory during times of severe memory pressure. Using UMA_ZONE_NOTRIM rather than UMA_ZONE_UNMANAGED is an attempt to keep this zone more reactive in the face of severe memory pressure.

Test Plan

Verify using vmstat -z that the ktls buffer zone does not shrink during off-peak hours on an otherwise busy web server, or when setting sysctl debug.uma_reclaim=3. (UMA_RECLAIM_TRIM)

Verify using vmstat -z that the ktls buffer zone does shrink if doing sysctl debug.uma_reclaim=2 or sysctl debug.uma_reclaim=1
(UMA_RECLAIM_DRAIN*)

[jhb]

This seems sensible to me.

[mav]

I understand the desire to avoid extra work, but if server load pattern really change to no longer require the KTLS, the allocations will stay cached, occupying memory for no reason if there is no explicit pressure. For example, considering ZFS might shrink ARC or at least limit its growth if there is no free memory, it might be a waste.