tree(3): allow the compare function to return any signed type

This allows to write very short comparison function when we are
comparing just pointer values:

return ((intptr_t)((uintptr_t)a->ptr/2 - (uintptr_t)b->ptr/2));

Reviewed by: dougm, alc
Differential revision: https://reviews.freebsd.org/D35722

(cherry picked from commit 86cdadbed44ecd36469dfbc4f1e492dfe51fcd4e)