[tex4ht] [bug #345] regression testing is needed

[Michal Hoftich]

Hi Deimi,

> 
> yes, I misunderstood you. Sorry about that.

That's fine :)

> However, starting from the scratch may be not so bad. 
> The system from  scratch should be more clean that 
> tweaked. (l3build can always serve us  as a reference)
> But first we should  understand our needs, and maybe 
> some solutions (I still don't know how to deal with multiple outputs)
> I need to reflect about it.

Maybe the process could be following:

- it should be possible to save the state of correct test run
  automatically
  - we can get list of all generated files from the .lg file
  - save these files in some subdir
  - save the file list in a file
- before test run, delete all generated files - we can get them from the
  .lg file + temporary, aux, dvi
- run the compilation 
  - we can alter $TEXINPUTS to include the trunk/lit
    directory, so we don't need to copy the .4ht files to the test dir.
  - ht-fonts dir can be passed to tex4ht with -i option, so it should be
    possible to test also .htf files. But I haven't tried that
  - use make4ht for the compilation - we can use build files to run
    biber, xindy, makeglossaries etc.
  - we don't need to use directly make4ht executable, but use it as a
    library, so we can incorporate it directly to the testing script
- test the log files for the errors
  - stop if they are there
- test the generated files
  - test the existence of all files that should be created
  - test that no other files are referenced in the .lg file
  - use diff for the text files
  - use imagemagick's compare for the pictures
  - we can run specific tests on files with some extension
    (validator for HTML?)
  - we can test using regexes
  - we can test using XML tools
  - generally, only the diff test is really needed?
- if there are differences between the saved and actual states which are
  on purpose, save the new state

So this is what I am thinking at the moment, not all of that is really
needed, but the tool should be flexible enough to enable that :)

Here is a sample script, it will get the current $TEXINPUTS variable,
include `tmp` subdirectory of the actual dir at the beginning and
compile the LaTeX document the new value of $TEXINPUTS. Note that it
also solves the issue that path separator in $TEXINPUTS is different on
Unix systems and on Windows (at least I think so)


     kpse.set_program_name "luatex"
     local dir = lfs.currentdir()
     local separator = os.type == "unix"  and ":" or ";"
     local stydir = dir .. "/tmp/"
     
     local texinputs = table.concat({stydir, kpse.expand_var("$TEXINPUTS")}, separator)
     print(stydir, ":"..texinputs..":")
     
     os.execute(string.format('export TEXINPUTS=%s && pdflatex sample ', texinputs))

Best regards,
Michal