Somewhat surprising behaviour of \scantokens after \endinput

[ud.usenetcorrespondence at web.de]

jfbu wrote:

> I have encountered a phenomenon with \scantokens executed after \endinput:
>
> ---
> % execute this file testscan.tex with etex on command line
> \def\test{%
> \begingroup
> \endlinechar-1\everyeof{\noexpand}%
> \edef\foo{\scantokens{}}%
> \endgroup
> }%
> \test % ok
> % \test\endinput % would be ok
> \endinput\test % not ok
> ---
>
> executing "etex testscan" (or pdftex, xetex, luatex, eptex) gives this:
>
> Runaway definition?
> ->
> ! File ended while scanning definition of \foo.
> <inserted text>
> }
> \test ...of {\noexpand }\edef \foo {\scantokens {}
> }\endgroup
> l.12 \endinput\test
> % not ok^^M
> ? X
>
> I would be curious if some expert could elucidate.


I don't know if I can be called an expert, but the man who wrote
the TeXbook surely is. ;-)

Chapter 20: Definitions (also called macros) of the TeXbook
has a very large double-dangerous-bend paragraph which
begins with

| Now let’s consider the control sequences that are expanded whenever
| expansion has not been inhibited. Such control sequences fall into
| several classes:

About \endinput you find:

| - \endinput. The expansion is null. The next time TeX gets to the end
| of an \input line, it will stop reading from the file containing that line.


What is specified by the phrase "The next time TeX gets to the end of an
\input line" is neither a specific "\input file" nor a specific
"\input line" but is just a moment in time when some condition is met.


Therefore I *think* expecting *some* error-message with your example
does correspond to the TeXbook's description of what behavior of TeX
is triggered by \endinput -- the reason for this expectation is:

\test expands to an \edef whose <definition text> contains \scantokens.

With \endinput\test the next time TeX gets to the end of an
\input line is "inside the \edef" while carrying out \scantokens as
\scantokens in turn leads to emulating
- unexpanded writing tokens to external text file and then
- processing that external text file via \input outgoing from current
catcode-régime.

(I just used the term "external text file".
Let's henceforth use the term "scantokens-file" for referring the same.)

Due to the previously encountered \endinput \scantokens' emulating
of processing that external text file via \input is stopped when TeX gets
to the end of the very first line of that "scantokens-file", i.e., is
stopped at a moment in time when gathering the <definition text> of an
\edef-assignment is still in progress and when the end of the
"scantokens-file" is not reached yet and therefore \everyeof is not
carried out.

This stopping in turn yields the runaway-definition-error.



Nonetheless I think the error-message encountered by you is
incorrect and misleading:

You find the statement:

   Runaway definition?
   ->
   ! File ended while scanning definition of \foo.

That statement is not correct in this subtle case.

In this subtle case the reason for the runaway-definition
is not that TeX encountered the _end_ of a file.

In this subtle case the reason for the runaway-definition
is that due to the pending \endinput reading from the
"scantokens-file" was ceased when TeX got to the end of the
very first line of the "scantokens-file".


In this subtle case a correct statement would probably
be something like:

   Runaway definition?
   ->
   ! Due to pending \endinput reading from file was ceased
   ! when getting to the end of its first line while scanning
   ! definition of \foo.



You can produce similar misleading statements without
eTeX-extensions:

Assume a .tex-input-file FileA.tex contains:

   Line 1
   Line 2
   Line 3


Assume a .tex-input-file test.tex contains:

   \endinput\edef\foobar{\input FileA.tex }%
   \bye

When compiling via tex test.tex, then the resulting
error-message on the console contains:

   Runaway definition?
   ->Line 1
   ! File ended while scanning definition of \foobar.

although the reason for the "runaway definition" is not
encountering the _end_ of a file (the end of FileA.tex) but
the reason for the "runaway definition" is ceasing processing
that file (i.e. FileA.tex) due to pending \endinput when
TeX gets to the end of that file's first line.



I could not resist to submit a bug-report for the
2029 TeX tune-up to the tex-k-mailing list about the
misleading text of the error-message - Subject:
Error-message is expected but the text of the
error-message is incorrect.



Yours sincerely

Ulrich Diez