rangelock: Fix handling of trylocks

When inserting a queue entry, i.e., locking a range, there are two
points where a trylock operation may fail, one before the new entry is
inserted, one after. In the latter case, rl_(r|w)_validate() would mark
the entry and rangelock_lock_int() would free it. However, this is of
course incorrect, since the entry is visible to other threads, which
will eventually attempt to remove it and free it again.

Factor out conflict handling in rl_(r|w)_validate() to a common function
as they are functionally the same. Then, introduce a new result which
indicates that a trylock failed but that the queue entry must not be
cleaned up.

While here, assert that a conflicting range isn't owned by the current
thread, as that would indicate a bug in the consumer.

Reviewed by: olce, kib
Reported by: syzkaller
Fixes: 5badbeeaf061 ("Re-implement rangelocks part 2")
Differential Revision: https://reviews.freebsd.org/D49438