vfs: try harder to find free vnodes when recycling
The free vnode marker can slide past eligible entries.
Artificially reducing vnode limit to 300k and spawning 104 workers each
creating a million files results in all of them trying to recycle, which
often fails when it should not have to.
Because of the excessive traffic in this scenario, the trylock to
requeue is virtually guaranteed to fail, meaning nothing gets pushed
forward.
Since no vnodes were found, the most unfortunate sleep for 1 second is
induced (see vn_alloc_hard, the "vlruwk" msleep).
Without the fix the machine is mostly idle with almost everyone stuck
off CPU waiting for the sleep to finish. With the fix it is busy
creating files.
Unrelated to the above problem the marker could have landed in a
similarly problematic spot for because of any failure in vtryrecycle.
Originally reported as poudriere builders stalling in a vnode-count
restricted setup.
Fixes: 138a5dafba31 ("vfs: trylock vnode requeue")
Reported by: Mark Millard
(cherry picked from commit c1d85ac3df82df721e3d33b292579c4de491488e)