kern: fail getgroup and setgroup with negative int

Found using
https://github.com/NetBSD/src/blob/trunk/tests/lib/libc/sys/t_getgroups.c

getgroups/setgroups want an int and therefore casting it to u_int
resulted in getgroups(-1, ...) not returning -1 / errno = EINVAL.

imp@ updated syscall.master and made changes markj@ suggested

PR: 189941
Tested by: imp@
Reviewed by: markj@
Pull Request: https://github.com/freebsd/freebsd-src/pull/407
Differential Revision: https://reviews.freebsd.org/D30617