vm: fix a number of functions to match the expected prototypes

Noticed while attempting to make boolean_t unsigned: some vm-related
function declarations and defintions were using boolean_t where they
should have used int, and vice versa.

MFC after: 1 week
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D39753

(cherry picked from commit f74be55e3045793a507d98127faf9aabd918e1ce)