cam: Drop periph lock when completing I/O with ENOMEM status

[imp]

Summary

When biofinish calls g_io_deliver with an error of ENOMEM, that kicks
off the slowdown protocol, forcing I/O to go through g_down rather than
be directly dispatch. One of the side effects is that the I/O is
resubmitted, so the start routines get called recursively, leading to a
recursive lock panic. Rather than make the periph lock recursive, drop
and reacquire the lock around such calls to biofinish.

For nda, this happens only when we can't allocate space to construct a
TRIM. For ada and da, this is only for certain ZONE operations.

Sponsored by: Netflix

[gallatin]

sys/cam/nvme/nvme_da.c

1087

I don't know the code well, but dropping a lock can introduce races.. I've spent a too long chasing my tail on that in the network stack where I suspect a race around dropping lock in a bug i'm working on.

I assumed you considered making the periph lock a recursive lock (eg, MTX_RECURSE) ?

[imp]

sys/cam/nvme/nvme_da.c

1087

We already drop the lock around the call to the SIM action routine, which is the normal exit path. And the periph lock is used all over CAM... so I'm hesitant to contemplate that...

[gallatin]

sys/cam/nvme/nvme_da.c

1087

OK, no objections then. I don't really feel qualified to review more than this hunk..