[luatex] LuaTeX versus ConTeXT Mk IV oddness

[Philipp Gesang]

···<date: 2014-08-04, Monday>···<from: Joseph Wright>···

> Not sure which list to raise this one on, but hopefully there will be
> enough people knowledgeable here to help!
> 
> Trying to read some UTF-8 data, I find that a few chars seem to give
> unexpected results with ConTeXt MkIV comped to plain LuaTeX using the
> same engine version:
> 
>     \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi
>     \def\test#1#2\stop{\showtokens{#1:#2}}
>      \test և\stop
> 
> on my system gives
> 
> This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/W32TeX) (rev 4971)
>  restricted \write18 enabled.
> (./test.tex
> > և:.
> \test #1#2\stop ^^@-\showtokens {#1:#2}
> 
> l.3 \test և\stop
> 
> ?
> 
> with "luatex" but
> 
> mtx-context     | warning: synctex is enabled
> mtx-context     | run 1: luatex
> --fmt="C:/texlive/2014/texmf-var/luatex-cache/context/a86c089b384a3076dc514ba966a1fac9/formats/luatex/cont-en"
> --jobname="test"
> --lua="C:/texlive/2014/texmf-var/luatex-cache/context/a86c089b384a3076dc514ba966a1fac9/formats/luatex/cont-en.lui"
> --no-parse-first-line --synctex=1 --c:currentrun=1
> --c:fulljobname="./test.tex" --c:input="./test.tex" --c:kindofrun=1
> --c:maxnofruns=8 --c:synctex "cont-yes.mkiv"
> This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/W32TeX) (rev 4971)
>  \write18 enabled.
> open source     > 1 > 1 >
> C:/texlive/2014/texmf-dist/tex/context/base/cont-yes.mkiv
> 
> ConTeXt  ver: 2014.05.21 22:04 MKIV beta  fmt: 2014.7.1  int:
> english/english
> 
> system          > 'cont-new.mkiv' loaded
> open source     > 2 > 2 >
> C:/texlive/2014/texmf-dist/tex/context/base/cont-new.mkiv
> close source    > 2 > 2 >
> C:/texlive/2014/texmf-dist/tex/context/base/cont-new.mkiv
> system          > files > jobname 'test', input './test', result 'test'
> fonts           > latin modern fonts are not preloaded
> languages       > language 'en' is active
> open source     > 2 > 3 >
> //ueahome3/stfsci2/c227/data/NTProfile/Desktop/test/test.tex
> > ե:ւ
> 
> tex error       > error on line 3 in file
> //ueahome3/stfsci2/c227/data/NTProfile/Desktop/test/test.tex: ?
> 
> 
> \test #1#2\stop ^^@-\showtokens {#1:#2}
> 
> l.3 \test եւ\stop
> 
> 
> 1     \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi
> 2     \def\test#1#2\stop{\showtokens{#1:#2}}
> 3 >>  \test և\stop
> 
> 
> ?
> 
> using ConTeXT.
> 
> As you can see, the engines are the same and indeed I'm running exactly
> the same file (encoded UTF-8). The issue here is not just that the
> results are different but that I get a different number of tokens: in
> the real situation that's a big issue. I've not tracked down all of the
> codepoints this problem applies to, but there seem to be several, so
> it's not just this char.
> 
> Any clues one what is up?

Context overloads the file reader. If you comment out the utf
related “textfileactions” in char-fio.lua (remake the format
afterwards) you get this:

    ConTeXt  ver: 2014.07.30 10:31 MKIV beta  fmt: 2014.8.4  int: english/english
    system          > 'cont-new.mkiv' loaded
    open source     > 2 > 2 > /home/phg/context/tex/texmf-context/tex/context/base/cont-new.mkiv
    close source    > 2 > 2 > /home/phg/context/tex/texmf-context/tex/context/base/cont-new.mkiv
    system          > files > jobname 'textfileactions', input './textfileactions', result 'textfileactions'
    fonts           > latin modern fonts are not preloaded
    languages       > language 'en' is active
    open source     > 2 > 3 > /home/phg/src/context-dev/playground/textfileactions.tex
    > և:

    tex error       > error on line 3 in file /home/phg/src/context-dev/playground/textfileactions.tex: ?


    \test #1#2\stop ^^@-\showtokens {#1:#2}
                                        
    l.3 \test և\stop
                  

    1     \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi
    2     \def\test#1#2\stop{\showtokens{#1:#2}}
    3 >>  \test և\stop
    4     

There was a thread a while ago about this interfering with
Hebrew: http://www.ntg.nl/pipermail/ntg-context/2012/069168.html

I haven’t checked for a while but back then it proved difficult
to disable this preprocessing from within a document because they
were being applied before the document was read.

Best,
Philipp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://tug.org/pipermail/luatex/attachments/20140804/4a58977d/attachment-0001.bin>