catopen(3): align returned errors with IEEE Std 1003.1™-2024

• Invalid/non-existent/unable to use message catalog file should result in ENOENT, and not in EFTYPE.
• Added detection of several cases of wrong file format due to length [*].
• Update man page.

• Based on the original patch from PR.

PR: 172805
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D47413