nvmft: Defer datamove operations to a pool of taskqueue threads

[jhb]

Summary

Some block devices may request datamove operations from an ithread
context while holding locks. Queue datamove operations to a taskqueue
backed by a thread pool to safely permit blocking allocations, etc. in
datamove handling.

Sponsored by: Chelsio Communications

[jhb]

NB: The iSCSI frontend just uses M_NOWAIT in all its datamove handling instead, but that seems non-ideal to me as that adds lots of places to do error checking.

[jhb]

sys/dev/nvmf/controller/ctl_frontend_nvmf.c

604

The task queues for the CTL backends reuse control_softc->ctl_proc as the process to "hold" their taskqueue threads. Here we just create kthreads inside of the "kernel" process. It would be a simple change to move these into the "ctl" process which might be a cleaner accounting, but I'm not sure how gross of a layering violation it is for a frontend to access control_softc.

[asomers]

I don't know this area of code, but your changes LGTM. At least, I can't spot anything wrong with the locking or error handling.