[tex-live] General test suite for TeX-Live

[Ken Moffat]

On Sun, Jun 19, 2016 at 09:33:41PM +0000, Karl Berry wrote:

(replying to Uwe, I think)

> All I have to add is that my previous attempt at automated testing
> (years and years ago) is in ./Build/tests.  The directory names will
> give the idea:
> 
> ./Build/tests/
> Makefile	 dvi-latex0-small2e/   largefile/
> README		 dvi-latex1-sample2e/  pdf-context0-hello/
> asytestlibs.asy  dvi-latex2-pdfprim/   pdf-context4-select/
> checkdvi.pl*	 dvi-latex5-tugboat/   pdf-latex0-small2e/
> checkpdf.pl*	 dvi-tex0-story/       pdf-tex0-story/
> common.mak	 dvi-tex5-tugboat/     tryconcat/
> 
> My approach for PDF (DVI is easy) was to make images and compare them.
> Turned out to be impractical, not surprisingly.

Which has an impact on the points you've copied in below (thanks for
doing that - it always helps if links can be read at some point in
the future).
> 
> The advent of l3build and "reproducible builds" for PDF should make the
> idea much more viable.  -k
> 
> P.S.
> > https://piratenpad.de/p/TeXLiveTesting
> 
> I don't understand why you put ideas in some read-only temporary url.
> Here is the text you wrote, for reading/archival on the mailing list
> like everything else in the thread.
> 
> Thoughts regarding automated TeX LIVE Testing
> 
> Requirements
> 
> * must run on at least Mac OS, Windows and Linux, more platforms are appreciated
> * shall not require manual checks

Nice if you can achieve it.

> * shall be self contained


I don't think I understand what you mean - e.g. you mention using
ImageMagik which is a separate program, and probably there will be
other external programs.

> * shall test a majority of user scenarios (letters, reports, books, etc.)

In my limited experience (and to be honest, I'm dubious about
posting on this because my latex skills are so limited) the fun
comes from various things within the texsphere - letters, reports,
etc should be tested but they may be tangential to likely problems.

I have my own tests for Beyond Linuxfromscratch which I mentioned
the other day, but they are fairly minimal and do require manual
review (I adapted my xindy tests to make sure the index did get
created - but only after some changes in how _I_ built it caused
failures).  And they are definitely not useful in general although
they should work on any 'nix-ish system with bash and a PDF viewer.

> * many small tests may be helpful, bigger test scenarios must not be omitted (I, for example, ran a 600 pages dissertation as part of my testing)

I know that CPU power continues to increase, and tests can be run in
parallel if you can keep the details separate, but for repeated
testing smaller is probably better.

> * cannot test each and every scenario users may come across
> 
Indeed, that is why people have to test for themselves.

> 
> Design Ideas
> 
> * use python (as I know it best)
> * make use of its unit test facilities

For a proof of concept, whatever you are familiar with.

> * run a testcase by calling TeX engine n-times on specific file

This I do not understand.  I understand using the engine the
required number of times - and interspersing that with calls to
other parts of TeX such as xindy or asymptote - but I'm not at all
clear what you are proposing.

> * check if the process ended properly

I don't know about your preferred python (I try to avoid scripting
languages where whitespace is important ;) but when my own tests run
from a Makefile hang in TeX I have to key 'x' to stop.  And
sometimes TeX reports normal status but part of the test did not
work.

> * proper process ending does not mean that file is correct
> * Gather statistics on file: Size, number of pages, ...
> * Convert generated file to image and run imagemagick to collect statistical data. 

This goes back to what Karl wrote above - what guarantee is there
that the version of ImageMagick you use today will give the same
results as the next version you happen to use ?

Also, I have no experience in using ImageMagick to split a
multi-page PDF into individual page images, but I suspect there
might be a large overhead and perhaps other tools such as qpdf may
be better for that.

> * Compare statistical data to known result. If theshold > level ==> testcase failed.

I'm not a statistician, but I suspect that the degree of acceptable
variation will probably differ for each test.

Interesting ideas, and don't let me discourage you.

ĸen
-- 
                       War is Peace
                    Freedom is Slavery
                   Ignorance is Strength