[lltx] \scantokens differs between LaTeX and plain?

[Heiko Oberdiek]

On Tue, Nov 15, 2011 at 11:01:12PM +0100, Philipp Stephani wrote:

> 2011/11/15 Arno Trautmann <Arno.Trautmann at gmx.de>:
> >> The code isn't supposed to do anything meaningful except show a bug.
> >> I've tried (not much) to tinker with formats, removing this from
> >> LuaLaTeX and adding that to plain TeX to try to spot what made the
> >> difference, but I had no success.
> >
> > Same here. Seems to be a mystery ? maybe some special code Tako built in to
> > manipulate LaTeX?
> 
> This is caused by the differences in \endlinechar between Plain and
> LaTeX. You can trigger the issue using the \endlinechar in math mode.
> The following document breaks only with LuaTeX:
> 
> \newlinechar=`q
> \scantokens{$q$}
> \bye

Or without formats:

\catcode`\{=1
\catcode`\}=2
\ifx\directlua\undefined
\else
  \directlua{tex.enableprimitives('',{'scantokens'})}
\fi
   
\newlinechar=`q
\edef\x{\scantokens{aqb\noexpand}} 
\immediate\write16{\meaning\x}

\csname @@end\endcsname\end

As result of "pdftex --ini --etex" macro:->a b is printed.
"luatex --ini --etex" breaks with an error:

  Runaway definition?
  ->a 
  ! File ended while scanning definition of \x.
  <inserted text> 
                  }
  l.9 \edef\x{\scantokens{a
  b\noexpand}
                                       }
  ?

Yours sincerely
  Heiko Oberdiek