vm_pageout: Make vmd_oom a bool

No functional change intended.

Reviewed by: dougm, kib
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Modirum MDPay
Differential Revision: https://reviews.freebsd.org/D48376

(cherry picked from commit fe1165df4b776b14b21a04d2ef3fc4c46740c2f5)