Speed up *-old-* make targets by using sed instead of xargs

[emaste]

Summary

Targets like 'list-old-files' used "xargs -n1" to produce a list of one
file per line.  Using xargs resulted in one fork+exec for each argument,
resulting in rather long runtime.  Use sed to split the list instead.

[emaste]

On one machine make list-old-files took 30s wall clock time with xargs and less than 1s with sed.

[jlduran]

Is this an April fools' joke?, if not nice!

How does trperform? (something like tr -s "[:space:]" "\n" ?

[emaste]

In D34741#787412, @jlduran_gmail.com wrote:

How does trperform? (something like tr -s "[:space:]" "\n" ?

I didn't try it, but the tens of thousands of forks were responsible for the xargs slowness, and sed does (and tr would) avoid that. sed is already included in ITOOLS, so is going to be available in any context where this could be used

[dnelson_1901_yahoo.com]

I had made a similar change in my local tree, but used make's :ts flag instead. I guess the sed expression more portable, though:

@@ -3202,9 +3211,9 @@
 list-old-files: .PHONY
        @cd ${.CURDIR}; \
        ${MAKE} -f ${.CURDIR}/Makefile.inc1 ${.MAKEFLAGS} ${.TARGET} \
-           -V OLD_FILES -V "OLD_FILES:Musr/share/*.gz:R" \
-           -V "OLD_FILES:Mlib/*.so.*:S,^lib,usr/lib32," \
-           -V "OLD_FILES:Musr/lib/*:S,^usr/lib,usr/lib32," | xargs -n1 | sort
+           -V "OLD_FILES:ts\n" -V "OLD_FILES:Musr/share/*.gz:R:ts\n" \
+           -V "OLD_FILES:Mlib/*.so.*:S,^lib,usr/lib32,:ts\n" \
+           -V "OLD_FILES:Musr/lib/*:S,^usr/lib,usr/lib32,:ts\n" | sort
 
 delete-old-files: .PHONY
        @echo ">>> Removing old files (only deletes safe to delete libs)"

There's a similar speedup that can be done by avoiding the ... | while read var pattern. The shell has to read from pipes one character at a time, and the old-files list is almost 1MB. It's much faster to send to a temp file, which the shell will then read in blocksize chunks:

@@ -3215,7 +3224,8 @@ delete-old-files: .PHONY
 # the Makefile parser segfault.
        @exec 3<&0; \
        cd ${.CURDIR}; \
-       ${MAKE} -f ${.CURDIR}/Makefile.inc1 ${.MAKEFLAGS} list-old-files | \
+       OLDTMP=$$(mktemp -t old); \
+       ${MAKE} -f ${.CURDIR}/Makefile.inc1 ${.MAKEFLAGS} list-old-files > $$OLDTMP; \
        while read file; do \
                if [ -f "${DESTDIR}/$${file}" -o -L "${DESTDIR}/$${file}" ]; then \
                        chflags noschg "${DESTDIR}/$${file}" 2>/dev/null || true; \
@@ -3228,7 +3238,8 @@ delete-old-files: .PHONY
                              <&3; \
                  fi; \
                done; \
-       done
+       done < $$OLDTMP; \
+       rm -f $$OLDTMP
 # Remove catpages without corresponding manpages.
        @exec 3<&0; \
        find ${DESTDIR}/usr/share/man/cat* ! -type d 2>/dev/null | sort | \

[emaste]

CC @sjg
I suspect :ts\n is OK

[sjg]

sed is a win over xargs but using :ts\n would be better still.