cam/iosched: Log outlier latency events

[imp]

Summary

Log outlier latency events to devd. In addition to counting, this will
allow analysis of whether the problem is confined to a specific block
range, or if it's a more general problem.

Sponsored by: Netflix

[imp]

Misc updates with testing

[jhb]

Do you want a global knob to control if devd reports are enabled rather than just the counts?

[imp]

In D46036#1049518, @jhb wrote:

Do you want a global knob to control if devd reports are enabled rather than just the counts?

devd will handle them fairly efficiently, so my first inclination is to always report. Other events from the kernel don't have this throttling..... though if you're getting a lot of these reported, they are forwarded to other listeners at least, which might be a problem for some people (I learned yesterday that firefox is one such listener). The default is 500ms in the device, which even in congested spinning disk land is kinda horrible and is semi-naturally rate limiting. Steady state that's 2/s. Due to head of queue blocking effects, that can be big bursts once in a while (more likely on devices with big queue depths like NVMe, though). We have a ~1k even buffer in the kernel. so they'll be generally rare, generally singletons, but on sick hardware can deviate from that. For that case, I'm not sure if you'd want to just increase this limit (because you know you have sick hardware and better on the way) or if you'd want a way to not get them at all.

So I'm on the fence about this...