libc: Fix size range check in setvbuf

From enh at google.com via openbsd-tech mailing list via pfg@:

The existing test is wrong for LP64, where size_t has twice as many
relevant bits as int, not just one. (Found by inspection by
rprichard.)

(cherry picked from commit 9515313b26beb005a521aff2e6edd4d75cd010da)