[tlbuild] Problems with parallel make

[Ken Brown]

I apologize in advance for the length of this message.  The short 
version is that I have two problems, described near the beginning, and a 
(temporary?) workaround, described at the end.

I have always used parallel make when building TeX Live (-j13 on my main 
build machine), but this year I'm having two problems with it.  The 
first might or might not be specific to my (Cygwin) environment, but I 
think the second is not.

1. 'make -j13 world' hangs during the 'make install-strip' step, with 
several 'make' instances still running.  I have to kill them all, and 
then I can complete the build with a (non-parallel) 'make world'.  I 
don't see any clues in the build log as to what the problem is.

2. 'make -j13 world' produces errors at the 'make check' step in the 
texk/web2c directory.  This time the build log does show what the 
problem is:

---------------------------------------------------------------------
FAIL: xetexdir/wcfname
======================
[...]

perl $srcdir/tests/fn-generate.perl || exit 128
+ perl ../../../texk/web2c/tests/fn-generate.perl
mv fn*-utf8.tex xetests/; rm -f fn*.tex
+ mv fn-utf8.tex fn£¥µÆÇñß-utf8.tex fn±×÷§¶-utf8.tex fnさざ波-utf8.tex 
xetests/
+ rm -f fn±×÷§¶-utf8-tmp.tex fnさざ波-euc.tex fnさざ波-sjis.tex

[...]

fenc="utf8"
+ fenc=utf8
for doc in fn-$fenc fn£¥µÆÇñß-$fenc fnさざ波-$fenc; do

   echo '>>> Document:'$doc '  File Encoding:'$fenc
   ./xetex -ini -interaction nonstopmode -jobname=$doc --shell-escape 
$doc.tex >xetests/$doc-term.log || rc=1
   mv $doc.txt $doc.log fn*-tmp.tex xetests/
   diff xetests/$doc.txt $srcdir/tests/fn-utf8.txt || rc=2

done
+ for doc in fn-$fenc fn£¥µÆÇñß-$fenc fnさざ波-$fenc
+ echo '>>> Document:fn-utf8' '  File Encoding:utf8'
 >>> Document:fn-utf8   File Encoding:utf8
+ ./xetex -ini -interaction nonstopmode -jobname=fn-utf8 --shell-escape 
fn-utf8.tex
cat: fn±×÷§¶-utf8-tmp.tex: No such file or directory
--------------------------------------------------------------------

So xetex is complaining about the non-existence of a file that was 
removed earlier.

I haven't analyzed this completely, but I think the problem is that the 
wcfname tests in various subdirectories of web2c are interfering with 
one another, with one test deleting files needed by another.

I don't know whether it's worth the trouble to serialize these tests so 
that this doesn't happen.  The Makefile is complicated enough that I 
didn't see an easy way to do that.  For the moment, I'm working around 
the problem by making sure that the '-j13' passed to the top level 'make 
world' doesn't propagate to 'make install-strip' and 'make check':

--- ../trunk/Build/source/Makefile.in   2023-02-08 11:45:39.582251400 -0500
+++ ./Makefile.in       2023-02-12 13:29:15.650600300 -0500
@@ -1035,9 +1035,10 @@
  # This (world) is the default target for the ./Build script in TeX Live.
  .PHONY: world
  world: all
-# Serialize 'make all' and 'make install-strip'.
+# Serialize 'make all' and 'make install-strip'.  Don't use parallel
+# make for latter.
         @echo "top-level make $@: running install-strip..."
-       $(MAKE) $(AM_MAKEFLAGS) install-strip
+       $(MAKE) $(AM_MAKEFLAGS) MAKEFLAGS= install-strip
  #
  # Just in case it's not a full checkout.
         if test -d "$(texlinks_dir)"; then \
@@ -1046,7 +1047,7 @@
         else :; fi
  #
         @echo "top-level make $@: running $(check_target)..."
-       $(MAKE) $(AM_MAKEFLAGS) $(check_target)
+       $(MAKE) $(AM_MAKEFLAGS) MAKEFLAGS= $(check_target)
  #
         @echo "top-level make $@ done."

Ken

P.S. I'm not necessarily suggesting that something like this should be 
done on the TL sources.  This is just my workaround.  If this were to be 
done on the sources, then it's Makefile.am that should be patched, not 
Makefile.in.