[tex-k] implementation of \scantokens

[Hendri Adriaens]

Hi all,

I have a small comment on the implementation of \scantokens. Let me
stress that I didn't look at the source, but found this when
programming some algorithm.

Context:
I was programming a method to read material verbatim and write it to
file for later use. It should work in the environment of powerdot
output from lyx, which looks like
\lyxbegin
...
\lyxend\lyxbegin
...
\lyxend

\lyxbegin should start reading verbatim, until it meets \lyxend. Then
switch to normal mode and execute anything that is left on the line.
The easiest is to read entire lines. But then the second \lyxbegin
should be read again to assign proper catcodes for execution and start
a new verbatim session. This seems to be the perfect example for the
use of \scantokens. However, it doesn't work here as that works with
an internal file operation, meaning that also an eof will enter the
input stream and the verbatim macros can't handle that.

The examples below demonstrate this problem. The first example
attempts an \input, which fails when parsing the line with the eof.
The second attempts \scantokens, which we might expect to work here,
but it doesn't as described above. The third example provides a very
cumbersome but working alternative (conditionally on the fact that we
know that we only wrote a single line that should be executed!).

Question:
Can't \scantokens be implemented to really put material back in TeX's
mouth for new catcode assignments? Of course, it will be a bit slower,
but then these unpleasant things won't happen anymore.

Thanks, best regards,
-Hendri.

Example 1:
\documentclass{article}
\makeatletter
\begin{document}
\toks@{\begin{verbatim}}
\immediate\openout\@inputcheck\jobname.tmp\relax
\immediate\write\@inputcheck{\the\toks@}
\immediate\closeout\@inputcheck
\input{\jobname.tmp}
test
\end{verbatim}
\end{document}

Example 2:
\documentclass{article}
\begin{document}
\scantokens{\begin{verbatim}}
test
\end{verbatim}
\end{document}

Example 3:
\documentclass{article}
\makeatletter
\begin{document}
\toks@{\begin{verbatim}}
\immediate\openout\@inputcheck\jobname.tmp\relax
\immediate\write\@inputcheck{\the\toks@}
\immediate\closeout\@inputcheck
\immediate\openin\@inputcheck\jobname.tmp\relax
\immediate\read\@inputcheck to\@tempa
\immediate\closein\@inputcheck
\@tempa
test
\end{verbatim}
\end{document}