versioning TeXlive packages on CTAN

[Brook Milligan]

> On Aug 24, 2019, at 8:16 PM, Norbert Preining <preining at logic.at> wrote:
> 
> On Sat, 24 Aug 2019, Brook Milligan wrote:
>> In the process of integrating TeXlive packages from CTAN into pkgsrc,
>> I (and others) run into a persistent issue that creates difficulties.
>> The main issue has two aspects: (i) TeXlive packages on CTAN do not
>> reflect version information in the filenames, and (ii) the contents of
>> the *.tar.xz files change periodically as a result of updates to the packages.
> 
> Yes, we are well aware of that problem for those distributions that do
> not provide their own versions of the original sources (like Debian,
> Ubuntu, RedHat etc), but rely on getting files that upstream provides.
> We have had this discussion several times here, or on tl-distribution
> mailing list.
> 
> Just recently on the FROSCON in Bonn I was discussing this with a
> colleague.
> 
> You are proposing to add the version number into the filename, and add
> links. This is indeed possible, but it will not help you (your users) in
> any way, because the versions you have registered in the pkgsrc with
> checksums might have already disappeared, making it impossible to
> install the package.
> 
> Before we discuss any further steps, how do you plan/would you deal with that?

Thanks for responding.  I am glad you are discussing these issues.

I cannot speak to other packaging systems, but that is no problem for pkgsrc.  Pkgsrc already maintains copies of all the distfiles in use, including archived versions that are no longer available on the original distribution sites.  In fact, that already happens for TeXlive packages. However, the process of getting them there is much more cumbersome that it would be if the TeXlive packages were versioned in the filenames.

Here is how things would work for us.

Initially, consider a hypothetical TeXlive package "foo" at version 1.2.3.  I am suggesting that it be visible on CTAN in the following way:

	foo-1.2.3.tar.xz (actual package file)
	foo.tar.xz -> foo-1.2.3.tar.xz

Now consider what happens with pkgsrc.

- The first time a pkgsrc bulk build happens (which is more or less daily), it will download foo-1.2.3.tar.xz from CTAN and verify the checksum.  Because it will pass (the URL to foo-1.2.3.tar.xz and the expected checksum is encoded in the package), the file foo-1.2.3.tar.xz will be archived "forever" on pkgsrc servers.

- Any users of pkgsrc would also download the same file from CTAN, the checksum will pass, and the package will be made as usual.  Users may or may not archive their distfiles, but they need not because pkgsrc maintains a copy independently of CTAN.

Now, suppose there is a new release of the package "foo" and the version is advanced to 1.2.4.  Suppose also that you do not wish to archive old versions; I get that that is infeasible.  Now CTAN looks like this:

	foo-1.2.4.tar.xz (actual package file, new version)
	foo.tar.xz -> foo.1.2.4.tar.xz

- If pkgsrc updates the package, the case is exactly as before.

- If pkgsrc does not update the package (so it is still pointing to foo-1.2.3.tar.xz), then users will be requesting a file that no longer exists on CTAN, the request will fail, and it will failover to the pkgsrc archive which still has the expected file.  Consequently, the download will succeed, the checksum will pass, and the package made.  Other than the extra fetch request, there is no difference for users compared to the previous case.  Either way, the package builds just fine.

Note that there is no burden on CTAN to make this work, other than a 404 response to requests when CTAN and pkgsrc do not agree on the version.  Users are not affected as there will always be another site (i.e., pkgsrc) that has the correct files so they will always be able to build their packages.

Note also that the unversioned file, e.g., foo.tar.xz, is entirely irrelevant in this situation.  I only suggest that you keep it to reduce surprises for any other people or systems that have been built to expect it.  The only cost of that is a symlink directory entry.

Finally, note that my example of a three field version number (e.g., 1.2.3) is also entirely irrelevant.  I know that many TeXlive packages have other versioning schemes.  Every package can use whatever form the developers wish, as long as it is a reasonable monotonically increasing sequence; all the ones I am familiar with already do this, so there is no problem there.  Thus, packages might be a hodgepodge of foo-1.2.3.tar.xz, bar-20160722.tar.xz, baz-7_2a.tar.xz, etc.  No problem; no imposition on current practice.

Thus, I think there is an entirely feasible solution to the problem that pkgsrc (and likely others) are facing that will not add burdens to you or CTAN as far as I can tell.

I hope you will implement this, because it would make a _huge_ difference to the usability of the TeXlive archives.

Thank you very much for considering this.

Cheers,
Brook