nfscl: Always invalidate buffers for append writes

kib@ reported a problem which was resolved by
reverting commit 867c27c23a5c, which changed the NFS
client to use direct RPCs to the server for
IO_APPEND writes. He also spotted that the
code only invalidated buffer cache buffers
when they were marked NMODIFIED (had been
written into).

This patch modifies the NFS VOP_WRITE() to
always invalidate the buffer cache buffers
and pages for the file when IO_APPEND is
specified. It also includes some cleanup
suggested by kib@.

Reported by: kib
Tested by: kib
Reviewed by: kib
MFC after: 10 weeks