[luatex] LuaTeX garbles special tokens in terminal output

[Henri Menke]

On 12/24/18 9:40 PM, Joseph Wright wrote:
> On 24/12/2018 08:34, Henri Menke wrote:
>> Dear devs,
>>
>> LuaTeX somehow garbles special tokens when printing to the log using
>> \show.  For example when you define a macro with active end of lines,
>> I'd expect ^^M to be printed in the log.  However, LuaTeX replaces ^^M
>> with \r, mashing all the lines together into one.
>>
>> Consider the follwing MWE:
>>
>> \begingroup
>> \obeylines
>> \gdef\lines{
>>    Hello LuaTeX!
>>
>>    Why do you garble newlines in
>>    terminal output?
>> }
>> \endgroup
>>
>> \show\lines
>>
>> \bye
>>
>> ------------------------------------------------
>>
>> Here is the log with pdfTeX.
>>
>> ------------------------------------------------
>>
>> $ pdftex test.tex
>> This is pdfTeX, Version 3.14159265-2.6-1.40.19 (TeX Live 2018)
>> (preloaded format=pdftex)
>>   restricted \write18 enabled.
>> entering extended mode
>> (./test.tex
>>> \lines=macro:
>> ->^^MHello LuaTeX!^^M^^MWhy do you garble newlines in^^Mterminal
>> output?^^M.
>> l.11 \show\lines
>>
>> ?
>>   )
>> No pages of output.
>> Transcript written on test.log.
>>
>> ------------------------------------------------
>>
>> And here is the log with LuaTeX.
>>
>> ------------------------------------------------
>>
>> $ luatex test.tex
>> This is LuaTeX, Version 1.09.0 (TeX Live 2019/dev)
>>   restricted system commands enabled.
>> (./test.tex
>>> \lines=macro:
>> .erminal output?e newlines in
>> l.11 \show\lines
>>
>> ?
>> )
>> warning  (pdf backend): no pages of output.
>> Transcript written on test.log.
>>
>> ------------------------------------------------
>>
>> Cheers, Henri
>>
> 
> This is by-design: see page 12 of the current manual:
> 
> Output to the terminal uses ^^ notation for the lower control range (𝑐
> < 32), with the exception
> of ^^I, ^^J and ^^M. These are considered ‘safe’ and therefore printed
> as-is.

Ahhh, that's really inconvenient.  This makes debugging macros handling
verbatim material impossible.  Is there a possibility to access the
signature and replacement text of a macro from Lua?

> 
> For l3build (which uses log comparison), we do a sweep over the log to
> turn returns into ^^M to get 'closer' to pdfTeX for exactly this reason.
> 
> Joseph