fusefs: fix the "write-through" of write-through cacheing
Our fusefs(5) module supports three cache modes: uncached, write-through,
and write-back. However, the write-through mode (which is the default) has
never actually worked as its name suggests. Rather, it's always been more
like "write-around". It wrote directly, bypassing the cache. The cache
would only be populated by a subsequent read of the same data.
This commit fixes that problem. Now the write-through mode works as one
would expect: write(2) immediately adds data to the cache and then blocks
while the daemon processes the write operation.
A side effect of this change is that non-cache-block-aligned writes will now
incur a read-modify-write cycle of the cache block. The old behavior
(bypassing write cache entirely) can still be achieved by opening a file
with O_DIRECT.
PR: 237588
Sponsored by: The FreeBSD Foundation