sched_ule: Enable preemption of curthread in the load balancer

[markj]

Summary

The load balancer executes from statclock and periodically tries to move
threads among CPUs in order to balance load. It may move a thread to
the current CPU (the loader balancer always runs on CPU 0). When it
does so, it may need to schedule preemption of the interrupted thread.
Use sched_setpreempt() to do so, same as sched_add().

PR: 264867

Test Plan

Using schedgraph.d I saw a softclock thread sit on CPU 0's runqueue for 50ms while
a batch-priority thread was running. Some reading of the individual trace entries
showed that the softclock thread was enqueued while running the load balancer. I
haven't been able to reproduce the problem with this patch applied.

[mav]

Not to this change, but it would be good if the load balancer could somehow prefer to migrate long running threads, not some random interrupt threads it may spot by pure luck.

sys/kern/sched_ule.c

2646–2648

Just a cosmetics and not yours, but in this particular case variable initialization in declaration would make more sense to me to assert the thread lock more specifically.

[kib]

sys/kern/sched_ule.c

955

Where is an AST is scheduled?

[markj]

sys/kern/sched_ule.c

955

In sched_setpreempt(). Probably it should be "preemption" instead?

[markj]

Address review feedback.

[kib]

sys/kern/sched_ule.c

955

You mean setting td_owepreempt, which is definitely not an ast. Might be, finish the sentence as schedule a preemption by setting td_owepreempt for current thread.

[markj]

sys/kern/sched_ule.c

955

If the interrupted thread was in user mode, then preemption is triggered by setting TDF_NEEDRESCHED, which is handled in ast(). td_owepreempt is set as well, but this mechanism only affects kernel mode interruptions.

[jhb]

sys/kern/sched_ule.c

953