Cleanup for rtld_malloc.c.

• Remove dead and most likely rotten MALLOC_DEBUG, MSTAT, and RCHECK options.
• Remove unused headers.
• Remove one case of undefined behavior where left shift could overflow. It is impossible on practice for rtld and libthr consumer.

PR: 237577
Sponsored by: The FreeBSD Foundation
MFC after: 1 week