nvmf: Permit failing I/O requests while disconnected

[jhb]

Summary

Add a kern.nvmf.fail_on_disconnection sysctl similar to the
kern.iscsi.fail_on_disconnection sysctl. This causes pending I/O
requests to fail with an error if an association is disconnected
instead of requeueing to be retried once the association is
reconnected. As with iSCSI, the default is to queue and retry
operations.

Sponsored by: Chelsio Communications

[jhb]

sys/dev/nvmf/host/nvmf_sim.c

44

So this did not really do what I wanted. It caused the periphs to go away entirely. I could rescan after a reconnect perhaps, but that is a larger change. I tried using CAM_SCSI_IT_NEXUS_LOST. That does leave the periphs around while failing I/O's with EIO, but it's also way more verbose as basically the periphs retry the commands and give up after retries are exhausted. FWIW, iSCSI uses CAM_DEV_NOT_THERE, but it also rescans on reconnect. I'm not quite sure what the "right" approach is. Adding in a rescan on reconnect is probably cleaner? I have most of a rescan written already in the handling of an "active namespaces changed" AER.

[imp]

sys/dev/nvmf/host/nvmf_sim.c

44

DEV_NOT_THERE is the signal from the sim or periph drive that the drive had departed and isn't coming back. So I'm not surprised the upper layer periph goes away...

You are looking for a code that tells CAM to pass an error up to the upper layers immediately, right? One that won't be retried. You likely need to coordinate with the nda periph so it doesn't retry I think. We might want to hoist some of the retry logic out of dev/nvme too...

A lot depends on what you want this to do. Fail the mounts and invalidate? Or just send errors up and hope for the best until reconnect...

[jhb]

sys/dev/nvmf/host/nvmf_sim.c

44

I do think we want to fail and not retry. The use case I've used this for with iSCSI is if you use GEOM_MULTIPATH on top of two iSCSI connections and create multipath consumers on top of each pair of duplicate LUNs. Then when you kill one iSCSI connection the I/O requests to the top-level GEOM_MULTIPATH disks just fail over to the other. The more I think about this though, I do think nvmf needs to rescan on reconnect anyway as a remote namespace might have gone away or been added while we were disconnected. I think I will first solve that problem of rescanning on reconnect and then probably keep this version as-is after that which should work.

[imp]

sys/dev/nvmf/host/nvmf_sim.c

44

OK. Just for the record, since I looked it up,

CAM_BUSY or CAM_RESRC_UNAVAIL will cause a retry some time later (500ms for either, though that's a tunable).
CAM_AUTOSENSE_FAIL will cause an immediate failure with EIO, though that's not a great match conceptually (this isn't an autosense failure to report additional status about an error).
CAM_DEV_NOT_THERE will invalidate the device and return ENXIO.

CAM_SCSI_IT_NEXUS_LOST is a target mode status, so cam_periph_error treats it like a default error and maybe does a retry.

CAM_NO_NEXUS is the best matching error, but we'd need to add that where we process CAM_AUTOSENSE_FAIL to get the just fail behavior, or add it to the CAM_DEV_NOT_THERE case if you wanted it to go away (though I'd think CAM_NO_NEXUS has an implicit "I can't do this right now, but could in a while" vibe to it, while CAM_DEV_NOT_THERE says the SIM has flat out given up on this device.

cam_periph_error is missing half a dozen of the newer errors, alas.

[jhb]

After rebasing this on top of the series to rescan on reconnect, this now works as desired, so I will keep it at as CAM_DEV_NOT_THERE but push the rescan on reconnect fixes first.