rtld-elf: link libcompiler_rt on all architectures

[rlibby]

Summary

Statically link rtld-elf with libcompiler_rt on all architectures so
that we don't need to try to pick and choose the bits we need from it
for each architecture (we now leave that to the linker). Compilers may
emit calls to support functions in this library, but because of the use
of the linker flag -nostdlib for rtld's special needs, the library is
not linked as normal.

Previously we had two different solutions. On some architectures, we
were able to extract reimplementations of the necessary builtin
functions from our special build of libc. On ARM, we just linked
libcompiler_rt.

This is motivated by the same issue as D26199 and D27665, but should be
a simpler solution that applies to all architectures.

---

This patch applies on top of a revert of D27665, which would be
committed concurrently.

Test Plan

make tinderbox WITHOUT_KERNELS=

env MAKEOBJDIRPREFIX=/usr/obj/gcc9 CROSS_TOOLCHAIN=amd64-gcc9 make buildworld
cc -m32 /tmp/hello.c -o /tmp/hello32
/usr/obj/gcc9/usr/src/freebsd/amd64.amd64/libexec/rtld-elf32/ld-elf32.so.1 -- /tmp/hello32

[rlibby]

Actually include all the changes

[arichardson]

I'm not particularly keen on building all of compiler-rt again without SSP just for the 2-3 files that rtld needs. I'd be surprised if those functions even use SSP with -fstack-protector-strong. Are there any functions in compiler-rt that would benefit from SSP?

[arichardson]

In D27736#620002, @arichardson wrote:

I'm not particularly keen on building all of compiler-rt again without SSP just for the 2-3 files that rtld needs. I'd be surprised if those functions even use SSP with -fstack-protector-strong. Are there any functions in compiler-rt that would benefit from SSP?

In other words, can we just link compiler-rt and see if we get any missing SSP symbols? If we do need a non-SSP version, I'd prefer if we just built the umoddi3.c udivdi3.c qdivrem.c files for all architectures inside RTLD instead of building all of compiler-rt again. The build is slow enough as it is, and adding more variants of libraries makes it even slower.

Or we could just build libcompiler-rt.a without SSP? I'm clearly biased by working on CHERI, but I'm not convinced that SSP is particularly useful since it doesn't do anything against non-sequential overflows.

[rlibby]

In D27736#620002, @arichardson wrote:

I'm not particularly keen on building all of compiler-rt again without SSP just for the 2-3 files that rtld needs. I'd be surprised if those functions even use SSP with -fstack-protector-strong. Are there any functions in compiler-rt that would benefit from SSP?

I have the impression that -fstack-protector-strong emits code for basically any calls to functions with out-parameters so it may have wider reach than one would think. compiler_rt does have functions that get instrumented due to calling functions with out parameters--the mod functions that call divmod functions. In particular, these objects are instrumented across most architectures: moddi3.o, modti3.o, umodti3.o.

As for actually benefit... I am not sure. You and kib have expressed opinions that SSP is not very valuable. Yet, we are building almost all of userspace and the kernel with SSP.

In D27736#620004, @arichardson wrote:

In other words, can we just link compiler-rt and see if we get any missing SSP symbols?

Currently it does link without the special no-SSP libcompiler_rt build, but it would be liable to break with updates of the compilers, the builtin library, or rtld.

If we do need a non-SSP version, I'd prefer if we just built the umoddi3.c udivdi3.c qdivrem.c files for all architectures inside RTLD instead of building all of compiler-rt again.

I did start by just trying to pull source files out of libcompiler_rt, and as you know that was my previous solution for i386 with udivmoddi4 (D27665). When I went to fix up ARM it got more complicated as ARM refers to more functions and some of them are assembler source. Additionally, libcompiler_rt sources need some more warnings disabled (which is not a big deal, just makes it uglier).

(Incidentally, qdivrem is a libc thing, it's not a direct dependency.)

The build is slow enough as it is, and adding more variants of libraries makes it even slower.

I'm a little surprised by this. We are currently already building almost 500 libraries in buildworld (including variants, but before 32-bit compat which brings it to almost 900), and this is C and assembler code (i.e. not C++!).

Or we could just build libcompiler-rt.a without SSP? I'm clearly biased by working on CHERI, but I'm not convinced that SSP is particularly useful since it doesn't do anything against non-sequential overflows.

We could... the other no-SSP libraries are: libgcc_eh, libgcc_s, and libthr. However, I am not eager to be the person that removes this hardening feature from other code, even if its value may be dubious.

Regardless, I hear your feedback. Let me see if I can reformulate this in a more palatable way. I have some other stuff to take care of so it may be a few days.

[arichardson]

In D27736#620188, @rlibby wrote:

In D27736#620002, @arichardson wrote:

I'm not particularly keen on building all of compiler-rt again without SSP just for the 2-3 files that rtld needs. I'd be surprised if those functions even use SSP with -fstack-protector-strong. Are there any functions in compiler-rt that would benefit from SSP?

I have the impression that -fstack-protector-strong emits code for basically any calls to functions with out-parameters so it may have wider reach than one would think. compiler_rt does have functions that get instrumented due to calling functions with out parameters--the mod functions that call divmod functions. In particular, these objects are instrumented across most architectures: moddi3.o, modti3.o, umodti3.o.

As for actually benefit... I am not sure. You and kib have expressed opinions that SSP is not very valuable. Yet, we are building almost all of userspace and the kernel with SSP.

SSP can be useful in some cases, but I'd assume that most files here just write a single out-param so it shouldn't matter.
One option to remove the depedency on the SSP runtime functions could be to add __attribute__((no_stack_protector)) to e.g. __moddi3() since we know that __udivmoddi4() will never overflow the out-param.
Although it's probably better to add -fno-stack-protector for that file since it avoids patching upstream LLVM files.

In D27736#620004, @arichardson wrote:

In other words, can we just link compiler-rt and see if we get any missing SSP symbols?

Currently it does link without the special no-SSP libcompiler_rt build, but it would be liable to break with updates of the compilers, the builtin library, or rtld.

If we do need a non-SSP version, I'd prefer if we just built the umoddi3.c udivdi3.c qdivrem.c files for all architectures inside RTLD instead of building all of compiler-rt again.

I did start by just trying to pull source files out of libcompiler_rt, and as you know that was my previous solution for i386 with udivmoddi4 (D27665). When I went to fix up ARM it got more complicated as ARM refers to more functions and some of them are assembler source. Additionally, libcompiler_rt sources need some more warnings disabled (which is not a big deal, just makes it uglier).

(Incidentally, qdivrem is a libc thing, it's not a direct dependency.)

This sounds like we have to add lots of machine-depedent logic (and basically duplicate the compiler-rt makefiles in rtld), so I agree that linking compiler-rt is the better solution.

The build is slow enough as it is, and adding more variants of libraries makes it even slower.

I'm a little surprised by this. We are currently already building almost 500 libraries in buildworld (including variants, but before 32-bit compat which brings it to almost 900), and this is C and assembler code (i.e. not C++!).

You're right this won't add to the build time much overall. However, it's almost 150 files that are being built again (and it's during the mostly serial part of the build) so I'm just wondering if there are other solutions.

Or we could just build libcompiler-rt.a without SSP? I'm clearly biased by working on CHERI, but I'm not convinced that SSP is particularly useful since it doesn't do anything against non-sequential overflows.

We could... the other no-SSP libraries are: libgcc_eh, libgcc_s, and libthr. However, I am not eager to be the person that removes this hardening feature from other code, even if its value may be dubious.

Regardless, I hear your feedback. Let me see if I can reformulate this in a more palatable way. I have some other stuff to take care of so it may be a few days.

My preferred solution would be if we could just link compiler-rt (either by disabling SSP for all of compiler-rt or just for the files that are used in RTLD). But this is just my opinion and this patch is better than the current state, so I'm also fine with committing this and working on removing the _NOSSP archives later.

[rlibby]

Here's the alternate where we just cross our fingers and link libcompiler_rt (no surprises):
https://github.com/rlibby/freebsd/commit/f61af4c21d62280d2b6e9c360606cae781d8f316

@kib, @arichardson, do you prefer building and linking the no-SSP library as currently in the review, or just linking the normal compiler_rt as in the link above?

[kib]

I think we should change compiler-rt to not compile with -fstack-protector, and then link it into rtld.

I realized that there is a (strong) argument to make the library not compiled with the flag. It is linked statically into binaries, because compiler support functions are not port of the OS ABI. So if compiler-rt is compiled with SSP, it is impossible to build a binary without SSP.

[rlibby]

In D27736#620834, @kib wrote:

I think we should change compiler-rt to not compile with -fstack-protector, and then link it into rtld.

I realized that there is a (strong) argument to make the library not compiled with the flag. It is linked statically into binaries, because compiler support functions are not port of the OS ABI. So if compiler-rt is compiled with SSP, it is impossible to build a binary without SSP.

Okay, I buy that. I'll work on the comments and update the review.

[rlibby]

feedback: just link libcompiler_rt, which will now be no-SSP

[rlibby]

libexec/rtld-elf/arm/Makefile.inc

4–6

Something to note: I think the second half of this comment is not accurate, so I have dropped it and disregarded it ordering the LIBADD. I think this order wrt rtld-libc.a was in fact the case already.