[luatex] behavior of \write in LuaTeX differs from the one of PdfTeX/XeTeX

[jfbu]

Le 26 mars 2015 à 18:38, jfbu <jfbu at free.fr> a écrit :

> Le 26 mars 2015 à 18:23, Hans Hagen <pragma at wxs.nl> a écrit :
> 
>> On 3/26/2015 5:09 PM, jfbu wrote:
>> 
>>> ok, but I don't understand here your reference to \obeylines, as there was no \obeylines but a \catcode13=12 in my code, but maybe that's just a missing "if you use \obeylines.."
>> 
>> it saw in some posted link but didn't look too close into it as i don't run latex here
>> 
>> % http://tug.org/pipermail/lualatex-dev/2011-November/001376.html
> 
> this explains it!

just for the record the \obeylines in the latex code snippet from 
the link above was irrelevant to the problem and can be commented out. 

It sets the \catcode of 13 to active, but as \endlinechar is then set 
to 10, the \toks0 receives a character 10 token (of catcode 12 as 
per default in latex). 

The issue is that 10 is also the default \newlinechar in LaTeX. 

With pdftex at least, the \scantokens
is truly close to a \write (with no expansion) followed by an \input, 
and the \newlinechar matters at the stage of the \write. 

In the code snippet from the link above setting \newlinechar to anything 
but 10 has no issues, but if it is set to 10 (the default in LaTeX), the 
\scantokens of LuaLaTeX just drops whatever is inside the \scantokens
after the encountered token of character code = \the\newlinechar

The issue about 7bit or 8bit output seems irrelevant here, here is
a variant of the code snippet from the link above where I chose
at random character 17 to play the role of \newlinechar

(sorry I don't know ConTeXT hence I will use Plain (e)TeX )

\catcode 17 = 12
\mathcode17=\the\mathcode`A

\begingroup
    \endlinechar=17
    \global\toks0{$
                  =A$}%
\endgroup %

\scantokens\expandafter{\the\toks0\relax}

\newlinechar = 17 % PROBLEM WITH LUATEX, NOT WITH ETEX/PDFTEX/XETEX
\scantokens\expandafter{\the\toks0\relax}

\bye

Jean-François