[tlbuild] zstd as package compressor

[Johannes Hielscher]

Am Tue, 18 Feb 2020 16:14:13 +1300
schrieb Henri Menke <henrimenke at gmail.com>:
> On TeX.SX a question came up “Why texlive installation is so slow?”
> and Norbert answered mentioning several points [1].

A wise man once warned about premature optimisation.

Please get me wrong, zstd is a great tool with a lot of potential to
speed up things, up to double-digit percent regions. But unless we
*know* that xz is the bad guy (= have profiled tlmgr on the very
platforms where it is considered “slow”), adding new code and build
complexity may not be worth the hassle IMHO.

I witnessed the Archlinux transition from xz to zstd, and (already
starting from a very good baseline) noticed hardly any difference. As
far as I understood it, this was made mainly for the mirror/download
bandwidth saves.

And, in my process list tlmgr work consists of guesstimated 63% Perl
meditation, 30% I/O wait load, and only 7% decompression work.
Multi-threaded and AI-based decompressors won't make inefficient
high-level language constructs any faster. Plus, quicker decompression
will only let sluggish I/O capacities (I'm looking at you, RPi, and
your µSD/LAN speed records) shine through harder.

IMO, being conservative about working on one package install/update a
time is a good thing, and then we are stuck with not gaining a lot from
a faster decompressor.

If there were a thing where users REALLY could save time, I'd rather
discuss asynchronous downloads: Instead of just iterating the tlmgr
subroutine [download → backup → unpack]; first create the full list of
downloads, start a curl worker in the background, and exploit the
networking and file I/O multitasking of today's platforms (backing
up/installing new packages, while others are being downloaded at the
same time).
In fact, I'm not aware ATM of any other package management system that
does such a thing at all.

Btw, zstd would have to be (made) available on all platforms then, and,
e. g., it pulls in xz/lzma as a dependency for itself (at least its
Debian and Arch packages).


Best,
Johannes