netlink: use size_t through the allocation KPI

This fixes some signedness bugs and potential underflows. The length of
nl_buf is still limited by UINT_MAX and this is asserted now.

Reviewed by: melifaro
Differential Revision: https://reviews.freebsd.org/D47551