libzfs_core: simplify max_pipe_buffer(), return new max

We do still check and only optionally set because all F_SETPIPE_SZ
directives reallocate the pipe buffer

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rich Ercolani <rincebrain@gmail.com>
Reviewed-by: Paul Dagnelie <pcd@delphix.com>
Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Closes #13133