libc/qsort: Don't allow interposing recursive calls

[arichardson]

Summary

This causes problems when using ASAN with a runtime older than 12.0 since
the intercept does not expect qsort() to call itself using an interposable
function call. This results in infinite recursion and stack exhaustion
when a binary compiled with -fsanitize=address calls qsort.
See also https://bugs.llvm.org/show_bug.cgi?id=46832 and
https://reviews.llvm.org/D84509 (ASAN runtime patch).

To prevent this problem, this patch uses a static helper function
for the actual qsort() implementation. This prevents interposition and
allows for direct calls. As a nice side-effect, we can also move the
qsort_s checks to the top-level function and out of the recursive
calls.

[arichardson]

ping? Is this approach okay or should I add a static qsort_impl helper function instead?

[dim]

I'm unsure what the consequence is of making these symbols have protected visibility. Won't this give any other nasty side effects?

[arichardson]

• Use a static function instead
• Move the qsort_s checks out of recursive calls

[kib]

lib/libc/stdlib/qsort.c

106

Do you ever return non-zero from local_qsort()? If not, why not make it void?

qsort_s would return (0) then.

107

I think it would be useful for debugging to name these local_qsort() functions differently for each I_AM_QSORT_X

[arichardson]

lib/libc/stdlib/qsort.c

106

good catch, I forgot to change that after moving the qsort_s checks.

107

Yes that makes sense, will update.

[arichardson]

change return type and rename local fns

[kib]

lib/libc/stdlib/qsort.c

234

No need.