mixer(8): Improve error messsages and warnings

No functional change intended.

Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Reviewed by: dev_submerge.ch, imp
Differential Revision: https://reviews.freebsd.org/D43793

(cherry picked from commit 09ba07011a627e3ea05e0be8ee6db76fd2371540)