libpfctl: improve syncookie watermark calculation

Ensure that we always pass sane limits for the high and low watermark
values.
This is especially important if users do something silly, like set the
state limit to 1. In that case we wound up calculating 0/0 as a limit,
which gets rejected by the kernel.

While here also shift the calculation to use uint64_t, so we don't end
up with overflows (and subsequently higher low than high values) with
very large state limits.

Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D36497