add soshutdown() calls to the server side krpc whenever a socket is no longer usable

[rmacklem]

Summary

It has been reported that a Linux NFSv4.1 client mount to a
FreeBSD server can result in the FreeBSD end of the TCP
socket being stuck in CLOSE_WAIT.
I think this can happen because the soclose() for the socket
cannot happen until the back channel is re-assigned to another
TCP connection.

This patch adds soshutdown(..SHUT_WR) calls where the socket
is no longer usable.

The patch is simple, but the question is "does soshutdown(..SHUT_WR)
sound like the correct thing to do to get the TCP connection past
CLOSE_WAIT?

Test Plan

Tested by network partitioning a client from a server
for a few minutes to break the TCP connection.

For some tests, the soclose() calls were commented
out, so that soshutdown(..SHUT_WR) was all that
was being done and the TCP connections did go
away.

[kevans]

The idea seems generally sane to me, but I would definitely wait for tuexen to pitch in. My understanding is that shutdown() will either progress us somewhat quickly to FIN_WAIT_1 or LAST_ACK depending on our prior state, and the tcp state machine will thereafter deliver the final blow modulo freeing local resources.

[tuexen]

It is fine to call soshutdown(..., SHUT_WR), when you know that you don't want to write anymore to the socket, but you want to read from it. This will trigger a state transition to FINWAIT-1 and then FINWAIT-2, if you are in ESTABLISHED and to LAST-ACK and then to CLOSED when you are in CLOSE-WAIT.
Please note that I do not know anything about NFS. But you say you call soshutdown(..., SHUT_WR), when the socket is no longer usable. Based on that description, I'm wondering why you need to keep the socket around locally and can't close it.

[rmacklem]

In the server side kernel RPC, the socket (struct socket *) is in a
structure called SVCXPRT (normally pointed to by "xprt").
These structures a ref counted and the soclose() is done
when the ref. cnt goes to zero. My understanding is that
"struct socket *" is free'd by soclose() so this cannot be done
before the xprt ref. cnt goes to zero.

For NFSv4.1/4.2 there is something called a back channel
which means that a "xprt" is used for server->client RPCs,
although the TCP connection is established by the client
to the server.
--> This back channel holds a ref cnt on "xprt" until the

 client re-assigns it to a different TCP connection
 via an operation called BindConnectionToSession
 and the Linux client is not doing this soon enough,
it appears.

So, the soclose() is delayed, which is why I think the
TCP connection gets stuck in CLOSE_WAIT and that is
why I've added the soshutdown(..SHUT_WR) calls,
which can happen before the client gets around to
re-assigning the back channel.