Stop using lorder and ranlib when building libraries

[arichardson]

Summary

Use of ranlib or lorder is no longer necessary with current linkers
(probably anything newer than ~1990) and ar's ability to create an object
index and symbol table in the archive.
Currently the build system uses lorder+tsort to sort the .o files in
dependency order so that a single-pass linker can use them. However,
we can use the -s flag to ar to add an index to the .a file which makes
lorder unnecessary.
Running ar -s is equivalent to running ranlib afterwards, so we can also
skip the ranlib invocation.

Similarly, we don't have to pass the .o files for shared libraries in
dependency order since both ld.bfd and ld.lld will correctly resolve
references between the .o files.

This removes many fork()+execve calls for each library so should speed up
builds a bit. Additionally lorder.sh uses a regular expression that is not
supported by the macOS libc or glibc and results in many warnings when
cross-building (see D25989).

See also https://www.gnu.org/software/coreutils/manual/html_node/tsort-background.html
for some more background:

This whole procedure has been obsolete since about 1980, because Unix
archives now contain a symbol table (traditionally built by ranlib, now
generally built by ar itself), and the Unix linker uses the symbol table
to effectively make multiple passes over an archive file.

Or alternatively https://www.unix.com/man-page/osf1/1/lorder/:

The lorder command is essentially obsolete. Use the following command in
its place: % ar -ts file.a

This depends on D26042 since otherwise ld complains about duplicate .o files (the only useful thing that lorder did was to remove duplicates)

Test Plan

buildworld still succeeds (after applying D26042)

[emaste]

share/mk/bsd.lib.mk

255

Could we just add -s to ${ARFLAGS}?
It looks like this file and share/mk/bsd.lib.mk are the only consumers, and the latter would benefit from having an ar symbol table anyhow.

[arichardson]

Add -s to ARFLAGS instead

[emaste]

This LGTM but I'd like one of the other reviewers to confirm this is sensible.

[imp]

Probably closer to ~2000 for the cut-over date. Toolchains in the 90s did support indexing, but I ran into several cases where lorder helped even though it shouldn't have... That did stop being the case sometime in the early 2000s, though... At least for static libraries. Dynamic ones did indeed work from about the mid 1990's w/o ranlib/tsort/lorder. Linux's a.out libraries being the last one I recall where that was beneficial.

The main risk here is that it reorders all the .o's and may cause too long of a call/jump that didn't happen before... Not a case I think we need to seriously worry about, but the possibility reinforces the notion of a universe before this commit.

Quibbles aside, i like this. There's a minor downside to the ARFLAGS change, but not enough of one to worry about. It will increase disk usage in builds a bit, but only in edge cases. The clarity and simplicity of the resulting use more than make up for some folks with extremely old code-bases that might have to adjust, or those that have added it by hand to their make files. This risk suggests an exprun might be warranted.

[emaste]

[dim]

LGTM. We can even MFC this.

[arichardson]

In D26044#578026, @imp wrote:

Probably closer to ~2000 for the cut-over date. Toolchains in the 90s did support indexing, but I ran into several cases where lorder helped even though it shouldn't have... That did stop being the case sometime in the early 2000s, though... At least for static libraries. Dynamic ones did indeed work from about the mid 1990's w/o ranlib/tsort/lorder. Linux's a.out libraries being the last one I recall where that was beneficial.

The main risk here is that it reorders all the .o's and may cause too long of a call/jump that didn't happen before... Not a case I think we need to seriously worry about, but the possibility reinforces the notion of a universe before this commit.

Quibbles aside, i like this. There's a minor downside to the ARFLAGS change, but not enough of one to worry about. It will increase disk usage in builds a bit, but only in edge cases. The clarity and simplicity of the resulting use more than make up for some folks with extremely old code-bases that might have to adjust, or those that have added it by hand to their make files. This risk suggests an exprun might be warranted.

I think this is unlikely to break anything other than duplicated sources so I was planning to commit it on Monday. @imp is this okay or should I ask for an exprun?

[zeising]

In D26044#585469, @arichardson wrote:

I think this is unlikely to break anything other than duplicated sources so I was planning to commit it on Monday. @imp is this okay or should I ask for an exprun?

Considering the issues that has been in ports lately, and the discussions that followed, I think it's a good idea to do an exp-run just in case.

[imp]

In D26044#585469, @arichardson wrote:

In D26044#578026, @imp wrote:

Probably closer to ~2000 for the cut-over date. Toolchains in the 90s did support indexing, but I ran into several cases where lorder helped even though it shouldn't have... That did stop being the case sometime in the early 2000s, though... At least for static libraries. Dynamic ones did indeed work from about the mid 1990's w/o ranlib/tsort/lorder. Linux's a.out libraries being the last one I recall where that was beneficial.

The main risk here is that it reorders all the .o's and may cause too long of a call/jump that didn't happen before... Not a case I think we need to seriously worry about, but the possibility reinforces the notion of a universe before this commit.

Quibbles aside, i like this. There's a minor downside to the ARFLAGS change, but not enough of one to worry about. It will increase disk usage in builds a bit, but only in edge cases. The clarity and simplicity of the resulting use more than make up for some folks with extremely old code-bases that might have to adjust, or those that have added it by hand to their make files. This risk suggests an exprun might be warranted.

I think this is unlikely to break anything other than duplicated sources so I was planning to commit it on Monday. @imp is this okay or should I ask for an exprun?

My suggestion is an abundance of caution one. I don't think it will break anything. But the ports collection prove how poor my imagination is. You have the final call, of course. If I had to guess, I'd say you are risking breaking 5-10 ports. You can view the latest turmoil two ways: it's in the noise or better safe than sorry. Don't know the lead time, though. If it's long, I'd chance it. If it's a couple of days I'd wait for it. Though it's up to you

[arichardson]

I've checked the affected ports manually and found one problem and will submit a patch for that shortly.

I used the following approach:

Find ports that appear to use bsd.lib.mk: (git grep -l "<bsd.lib.mk>" | sed 's/\/files\/.*//' | sed 's/\/Makefile//' | sort | uniq)

archivers/libcabinet
archivers/liblzxcomp
comms/birda
comms/hcidump
comms/librs232
comms/mlan3
converters/iconv
databases/libnvpair
databases/tcl-Mysql
devel/cgilib
devel/jech-dht
devel/libdistance
devel/libgpc
devel/libpasori
devel/libs11n
devel/libshhmsg
devel/libshhopt
devel/libublio
devel/libxalloc
devel/lua-gettext
devel/lua-pty
devel/st
devel/trio
finance/libstocks
games/libshhcards
games/quake2-3zb2
games/quake2-ctf
games/quake2-data
games/quake2-rogue
games/quake2-source
games/quake2-xatrix
graphics/gdchart
graphics/gdtclft
graphics/libfpx
graphics/libjxr
graphics/libmng
graphics/pgplot
graphics/squish
graphics/tifmerge
java/jxgrabkey
java/netty
mail/faces
mail/libdkim
mail/libdomainkeys
math/eispack
math/levmar
math/libmissing
math/linpack
math/lrng
math/randlib
math/sfft
math/slatec
math/spblas
multimedia/ab25
net/libtnl
net/libutp
net/packetdrill
net/socketbind
net/udt
security/fbopenssl
security/ipv6toolkit
security/pam_pwdfile
textproc/asm-xml
textproc/libcsv
textproc/nunnimcax
textproc/tinyxml
textproc/wordnet
www/tclhttpd
x11-toolkits/itk

Building those ports I got the following failures both with and without this patch applied (most seem to be missing deps on my build machine):

comms/birda
databases/tcl-Mysql
devel/libdistance
devel/lua-gettext
devel/lua-pty
games/quake2-ctf
games/quake2-data
graphics/libmng
graphics/pgplot
java/jxgrabkey
java/netty
mail/libdkim
math/levmar
math/linpack
multimedia/ab25
security/fbopenssl
textproc/asm-xml
textproc/wordnet
x11-toolkits/itk

I used the following command to check whether the ports build:

for i in $(cat affected-ports); do
  (make SRC_BASE=~/devel/freebsd PORT_DBDIR=~/devel/freebsd-ports-db \
        INSTALL_AS_USER=1 BATCH=yes NO_PKG_REGISTER=yes DEV_WARNING_WAIT=0 \
        -m ~/devel/freebsd/share/mk -C $i clean build || echo $i >> broken-ports-with-patch.txt) |& tee _.${i//\//_}.log;
done

[zeising]

@arichardson I'm not sure that is enough, there might be ports that include bsd.lib.mk from Makefiles in their respective tarballs.

The number of ports affected might be smaller than I thought though, I thought this was used for basically anything that's being linked on FreeBSD, but bsd.lib.mk isn't used in the ports infra (in ports/Mk/), which I thought was the case.

[dim]

In D26044#586338, @zeising wrote:

@arichardson I'm not sure that is enough, there might be ports that include bsd.lib.mk from Makefiles in their respective tarballs.

The number of ports affected might be smaller than I thought though, I thought this was used for basically anything that's being linked on FreeBSD, but bsd.lib.mk isn't used in the ports infra (in ports/Mk/), which I thought was the case.

No, there are really a minor number of ports which use the "BSD way" to build, e.g. use a bmake/pmake compatible Makefile in combination with <bsd.*.mk>. But these are generally exceptions, I think pkg itself might be the only important one. :)

I'd say most (80%? :) of the ports tree uses autotools/GNU make, then CMake in combination with either make or ninja, and a minority uses more exotic stuff like waf, meson and/or bespoke build systems.

That said, it could very well be possible that some ports with autotools also attempt to do "smart sorting" when linking, but that is mostly out of our hands, I guess.