versioned container testing

[Dmitry Kalinkin]

On Wed, Nov 20, 2019 at 6:24 PM Karl Berry <karl at freefriends.org> wrote:
> Huh. No, it's not intentional. I guess I didn't insert the code to
> remove older containers. I don't think we can ask CTAN and all mirrors
> to distribute all versions of all TL packages :).

According to my estimate we are looking at increase from 3.8 Gb to
about 10 Gb per year. So this is only a factor of ~2.5 increase, not
entirely out of reach, I guess.

> Norbert is maintaining an archive of all TL package releases:
> https://texlive.info/tlnet-archive/
> E.g., for (my) today,
> https://texlive.info/tlnet-archive/2019/11/20/tlnet/ (archive/ subdir as usual)
>
> So maybe that will help you? It should be no problem to make it
> available by rsync as well if it matters. --thanks, karl.

I think this could actually be the solution for us. Can we get
permission to point our urls to this mirror? This should not cause a
big load for you because we have a front end cache done by
hydra.nixos.org, so most users will be using that. There will be only
a few who will get through. Looking at my server's statistics, I see
about 5 significant (in number of files) downloads per day. And, if
this ever causes a problem, you can easily block us, as we clearly
state "Nixpkgs" in our User-Agent.

The main problems that we are looking to solve with this are:
1) Reproducibility
Even though we have some caching infrastructure set up, we would like
for build recipes to work even if all of our infrastructure is down
(this could happen if AWS goes offline), so, ideally, all urls should
point to a valid upstream downloads that will be around for many years
or even indefinitely. We really appreciate that we can now very easily
get any software exactly the way it was in, say, a NixOS release from
2016. This is the main promise of our distro and we are looking to
ensure that it also holds for our texlive package.
2) Upgrades
Our users want latest texlive for some reason. I'm trying to put only
-final snapshots (like
tex/historic/systems/texlive/2018/tlnet-final/archive) into the stable
releases of NixOS to ensure maximal reproducibility. If we were to
only do that, our updates would always lag by a year. So we do some
intermediate updates, but those can only be performed by our community
members who can actually host the snapshot.
3) Security audit
Doing our own snapshots means people are essentially forced to trust
me or, otherwise, do some work to verify my tarballs. If we make the
switch, they would have to trust Norbert, and that should be a bit
better, perhaps.