Fix installation of libxo encoder modules.

Because the install location was hardcoded in the Makefile as
/usr/lib/libxo/encoder, the lib32 version was installed over the native
version. Replace /usr/lib with ${LIBDIR}.

Also define SHLIB_NAME instead of LIB + FILES. This prevents building a
static library.

MFC after: 2 weeks