[tex-live] uninitialized line number in pdfTeX

[Karl Berry]

Taco, Akira, and Peter when you're back, and anyone else who's interested,

    Date: Sat, 13 Aug 2011 07:11:24 +0200 (CEST)
    From: Werner LEMBERG <wl at gnu.org>
    ...
    >     The attached texinfo file produces this error message
    >       ./WarningProb.texi:-1162167622: Could not find image file [...]

(This bug turns out to be quite a pain!)
Evidently luatex inherited the problem as well, and (non-pdf)etex.  More
details about reproducing below.

Let me draw your attention to our print_file_line procedure which prints
a --file-line-error style message (in printing.w in luatex, tex.ch
otherwise).  It starts like this:

  level:=in_open;
  while (level>0) and (full_source_filename_stack[level]=0) do
    decr(level);

That is, finding the topmost full_source_filename that is present.
(In the problem case, the value for `level' turns out to be 1.)
Fine.

Now, the procedure continues by printing the line number as follows:

    if level=in_open then print_int (line)
    else print_int (line_stack[index+1-(in_open-level)]);

We are in the else case (`in_open' is 2).  My question is, why
isn't this line simply
    else print_int (line_stack[level+1]);
instead of the complicated expression?  We just determined the stack
level for the filename which we printed, and it is `level', so shouldn't
we print the line number from that same place?

I believe the +1 is needed because the current line number is in `line',
not part of `line_stack', while there is no corresponding variable for
full_source_filename.  (Pretty confusing, and I can't completely
convince myself that argument is correct, but it seems to work out that way.)

I tried a few error cases with line_stack[level+1] and it worked ok, but
of course that doesn't prove much :(.

I can't comprehend the meaning behind `index+1-(in_open-level)' even
though it obviously works in almost all cases.  It appears that
print_file_line has done it this way since its inception in web2c-7.5.3.

What ultimately happens in the original source is that print_file_line
refers to an uninitialized slot in line_stack, because
index+1-(in_open-level) is 5:
- index=5, which is evidently the correct number of current input states
  (see below).
- in_open is 2, which is evidently the correct index of the top state
  coming from a file.
- level is 1, which is evidently the correct index of the top
  full_source_filename_stack entry.
- so the result is 5+1 - (2-1) = 6 - 1 = 5.  But this is wrong; we must
  not refer to line_stack[5], as that is a token list without a line
  number.  The maximum valid line_stack index at this point is 2.

I can't really expect my verbal description to be comprehensible (I
wrote it out mostly to convince myself I had at least some grip on the
problem), but I hope you'll try it out in the debugger and see what you
think shuld be done.  Something has to give ...

Thanks,
karl

Appendix: how to reproduce the bug.  Here is the minimal source file
I've been able to find, call it wprob.tex:

-----------------------------------------------------
\input texinfo

@macro sourceimage{FILENAME,WIDTH,HEIGHT,ALTTEXT}
@image{\FILENAME\,\WIDTH\,\HEIGHT\,\ALTTEXT\}
@end macro

@sourceimage{nosuchimage,,,}

@bye
-----------------------------------------------------

Then, run with environment variables to make the failure reliable:
env MALLOC_CHECK_=3 MALLOC_PERTURB_=69 \
etex --file-line-error --interaction=nonstopmode wprob.tex

(Without setting the MALLOC_ envvars, or on non-glibc platforms, the
value might be garbage, or it might be 0, or who knows.)

What happens in texinfo.tex is that \scantokens is used.  (The failure
does not happen with original tex, without the \scantokens primitive --
texinfo.tex uses a temp file in this case.)  With etex, the actual
\errmessage comes from \epsfgetbb in epsf.tex, the context is something
like this (with several other macro levels):

\scantokens{... \ifeof ... \errmessage ...}

I spent a little time trying to reproduce the same context outside of
texinfo.tex, but couldn't do so, so just worked with it.  The result
with pdftex (PDF output) is similar, though epsf.tex is not involved.
Any reasonable texinfo.tex should do, certainly the one in TL now.  Here
is the exact transcript I am getting.

$ env MALLOC_CHECK_=3 MALLOC_PERTURB_=69 \
  etex --file-line-error wprob.tex </dev/null
malloc: using debugging hooks
This is pdfTeX, Version 3.1415926-2.3-1.40.12 (TeX Live 2011)
 restricted \write18 enabled.
entering extended mode
(./wprob.tex (./texinfo.tex Loading texinfo [version 2011-05-23.16]: pdf,
fonts, markup, glyphs, page headings, tables, conditionals, indexing,
sectioning, toc, environments, defuns, macros, cross references, insertions,
(/home/texlive/karl/Master/texmf-dist/tex/generic/epsf/epsf.tex
This is `epsf.tex' v2.7.4 <14 February 2011>
) localization, formatting, and turning on texinfo input format.)
(nosuchimage.eps
./wprob.tex:-1162167622: Could not open file nosuchimage.eps, ignoring it.
@epsfgetbb ...Could not open file #1, ignoring it}
                                                  @else {@chardef @other = 1...

@next #1->@epsfgetbb {#1}
                         @epsfsetgraph {#1}
@imagexxx ...ysize =#3 at relax @fi @epsfbox {#1.eps}
                                                  @fi @ifimagevmode @medskip...

@image ...true @fi @else @imagexxx #1,,,,, at finish
                                                  @fi
l.1 ...eatspaces {}, at xeatspaces {}, at xeatspaces {}}
                                                  @empty
@scanmacro ...spaceisspace @scantokens {#1 at empty }
                                                  @endgroup
..
l.7 @sourceimage{nosuchimage,,,}

?