Under massive connection thrashing (web server restarting), we see long periods where the web server blocks when enabling ktls offload when NIC ktls offload is enabled.
It turns out the driver uses a single-threaded linux work queue to serialize the commands that must be sent to the nic to allocate and free tls resources. When freeing sessions, this work is handled asynchronously. However, when allocating sessions, the work is handled synchronously and the driver waits for the work to complete before returning. When under massive connection thrashing, the work queue is first filled by TLS sessions closing. Then when new sessions arrive, the web server enables kTLS and blocks while the tens or hundreds of thousands of sessions closes queued up are processed by the NIC.
Rather than using the work queue to open a TLS session on the NIC, switch to doing the open directly. This allows use to cut in front of all those sessions that are waiting to close, and minimize the amount of time the web server blocks. The risk is that the NIC may be out of resources because it has not processed all of those session frees. So if we fail to open a session directly, we fall back to using the work queue.