umtx: shm: Prevent reference counting overflow

This hardens against provoked use-after-free occurences should there be
reference counting leaks in the future (which is currently not the
case).

At the deepest level, umtx_shm_find_reg_unlocked() now returns EOVERFLOW
when it cannot grant an additional reference to the registry object, and
so will umtx_shm_find_reg(). umtx_shm_create_reg() will fail if calling
umtx_shm_find_reg() returns EOVERFLOW (meaning a SHM object for the
passed key already exists, but we can't acquire another reference on
it), avoiding the creation of a duplicate registry entry for a given key
(this wouldn't pose problem for the rest of the code in its current
form, but is expressly avoided for intelligibility and hardening
purposes).

Since umtx_shm_find_reg*(), and consequently the whole _umtx_op() system
call, can only return EOVERFLOW on such a bug manifesting, we don't
document that return value.

Reviewed by: kib, emaste
Approved by: emaste (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46126

(cherry picked from commit c3e6dfe55c0e81d0717b0458bc95128384c3ebe8)
(cherry picked from commit b20ae160872071fc20e5dde27051792177057fa5)

Approved by: so