shm: Handle swap pager allocation failures

shm_alloc() can fail if swap reservation fails (i.e., vm.overcommit is
non-zero) or racct is imposing some limits on swap usage.

PR: 282994
MFC after: 2 weeks
Reviewed by: olce, kib
Differential Revision: https://reviews.freebsd.org/D47839

(cherry picked from commit f3b7dbdad53b31492757417fc1336ed74ec80fd8)