[luatex] \write16 (or 128) loosing tokens after null character

[Hans Hagen]

On 11/20/2015 10:45 AM, jfbu wrote:
>
> Le 20 nov. 2015 à 10:33, Hans Hagen <pragma at wxs.nl> a écrit :
>
>> On 11/20/2015 9:54 AM, jfbu wrote:
>>>
>>> Le 19 nov. 2015 à 23:44, jfbu <jfbu at free.fr> a écrit :
>>>
>>>> Hi,
>>>>
>>>> this might be the same issue than one with \scantokens
>>>> which was discussed some time ago on this list (lost the reference)
>>>
>>> this was
>>>
>>> http://tug.org/pipermail/luatex/2015-March/005138.html
>>> [luatex] behavior of \write in LuaTeX differs from the one of	PdfTeX/XeTeX
>>>
>>> which referenced some other threads,
>>>
>>> I tried also running pdftex with -8bit switch which
>>> was mentioned by Hans in the previous thread and this gives
>>>
>>> This is pdfTeX, Version 3.14159265-2.6-1.40.16 (TeX Live 2015) (preloaded format=pdftex 2015.10.14)  20 NOV 2015 09:50
>>> entering extended mode
>>>   restricted \write18 enabled.
>>>   %&-line parsing enabled.
>>> **testlualog.tex
>>> (./testlualog.tex macro:->\romannumeral ` foo bar foo bar foo bar
>>> macro:->\romannumeral ` foo bar foo bar foo bar
>>>   )
>>>
>>>   (contains two null characters)
>>>
>>> The log with luatex is:
>>>
>>> This is LuaTeX, Version beta-0.80.0 (TeX Live 2015) (rev 5238) (format=luatex 2015.10.14)  20 NOV 2015 09:53
>>>   restricted \write18 enabled.
>>> **\input testlualog.tex
>>> (./testlualog.tex macro:->\romannumeral ` foo bar foo bar foo bar
>>> macro:->\romannumeral `
>>> )
>>>
>>> only one null character in \message output
>>>
>>> in \write output the null character is not there
>>> (and the other things are gone)
>>> best,
>>
>> well, because we're in a mixed tex/lua environment 0's can  always be sort of an issue .. just try to avoid them .. the times that we needed those slots for fonts are gone (in fact: avoid chars < 32)
>>
>
> I see, but the null slot here is not used for typesetting but
> for TeX programming tricks. The specific issue I face can

use other tricks (or don't care about the log)

> be overcome by checking for Lua engine and doing extra things
> (the goal is to provide some info in the log, it is peripheral
> to the main package activities, but dropping it would mean
> I must add an explication in the user manual as well)
> but I am wondering:
>
> as LuaTeX enhances greatly all TeX abilities, does not that
> include processing of files which may legitimately contain
> ascii chars < 32 ? isn't it paradoxal if such non-typographical
> programming like manipulations of data files suffers some
> limitation of \write under LuaTeX ?

it enhances and also changes a few things .. if you want to manipulatev 
data in luatex use lua

> and why does \message behave differently than \write ?

different code paths

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
     tel: 038 477 53 69 | voip: 087 875 68 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------