x86: Add a required store-load barrier in cpu_idle()

[markj]

Summary

ULE's tdq_notify() tries to avoid delivering IPIs to the idle thread.
In particular, it tries to detect whether the idle thread is running.
There are two mechanisms for this:

- tdq_cpu_idle, an MI flag which is set prior to calling cpu_idle(). If
tdq_cpu_idle == 0, then no IPI is needed.
- idle_state, an x86-specific state flag which is updated after
cpu_idleclock() is called.

The implementation of the second mechanism is racy; the race can cause a
CPU to go to sleep with pending work. Specifically, cpu_idle_*() set
idle_state = STATE_SLEEPING, then check for pending work by loading the
tdq_load field of the CPU's runqueue. These operations can be
reordered so that the idle thread observes tdq_load == 0, and
tdq_notify() observes idle_state == STATE_RUNNING.

Some counters indicate that the idle_state check in tdq_notify()
frequently elides an IPI. So, fix the problem by inserting a fence
after the store to idle_state.

PR: 264867

[kib]

sys/x86/x86/cpu_machdep.c

552

Where is the dual fence?

[markj]

sys/x86/x86/cpu_machdep.c

552

It is in ULE's tdq_notify(). With this change, there are two fences in the idle path, one in sched_idletd(), one here. Some instrumentation indicated that both checks for load catch a significant number of cases.

[kib]

sys/x86/x86/cpu_machdep.c

552

I suggest to note this fact in the comment.

[markj]

Note that tdq_notify() requires a fence as well.