[luatex] LuaTeX versus ConTeXT Mk IV oddness

[Joseph Wright]

On 04/08/2014 15:50, Joseph Wright wrote:
> Hello all,
> 
> Not sure which list to raise this one on, but hopefully there will be
> enough people knowledgeable here to help!
> 
> Trying to read some UTF-8 data, I find that a few chars seem to give
> unexpected results with ConTeXt MkIV comped to plain LuaTeX using the
> same engine version:
> 
>     \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi
>     \def\test#1#2\stop{\showtokens{#1:#2}}
>      \test և\stop
> 
> on my system gives
> 
> This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/W32TeX) (rev 4971)
>  restricted \write18 enabled.
> (./test.tex
>> և:.
> \test #1#2\stop ^^@-\showtokens {#1:#2}
> 
> l.3 \test և\stop
> 
> ?
> 
> with "luatex" but
> 
> mtx-context     | warning: synctex is enabled
> mtx-context     | run 1: luatex
> --fmt="C:/texlive/2014/texmf-var/luatex-cache/context/a86c089b384a3076dc514ba966a1fac9/formats/luatex/cont-en"
> --jobname="test"
> --lua="C:/texlive/2014/texmf-var/luatex-cache/context/a86c089b384a3076dc514ba966a1fac9/formats/luatex/cont-en.lui"
> --no-parse-first-line --synctex=1 --c:currentrun=1
> --c:fulljobname="./test.tex" --c:input="./test.tex" --c:kindofrun=1
> --c:maxnofruns=8 --c:synctex "cont-yes.mkiv"
> This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/W32TeX) (rev 4971)
>  \write18 enabled.
> open source     > 1 > 1 >
> C:/texlive/2014/texmf-dist/tex/context/base/cont-yes.mkiv
> 
> ConTeXt  ver: 2014.05.21 22:04 MKIV beta  fmt: 2014.7.1  int:
> english/english
> 
> system          > 'cont-new.mkiv' loaded
> open source     > 2 > 2 >
> C:/texlive/2014/texmf-dist/tex/context/base/cont-new.mkiv
> close source    > 2 > 2 >
> C:/texlive/2014/texmf-dist/tex/context/base/cont-new.mkiv
> system          > files > jobname 'test', input './test', result 'test'
> fonts           > latin modern fonts are not preloaded
> languages       > language 'en' is active
> open source     > 2 > 3 >
> //ueahome3/stfsci2/c227/data/NTProfile/Desktop/test/test.tex
>> ե:ւ
> 
> tex error       > error on line 3 in file
> //ueahome3/stfsci2/c227/data/NTProfile/Desktop/test/test.tex: ?
> 
> 
> \test #1#2\stop ^^@-\showtokens {#1:#2}
> 
> l.3 \test եւ\stop
> 
> 
> 1     \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi
> 2     \def\test#1#2\stop{\showtokens{#1:#2}}
> 3 >>  \test և\stop
> 
> 
> ?
> 
> using ConTeXT.
> 
> As you can see, the engines are the same and indeed I'm running exactly
> the same file (encoded UTF-8). The issue here is not just that the
> results are different but that I get a different number of tokens: in
> the real situation that's a big issue. I've not tracked down all of the
> codepoints this problem applies to, but there seem to be several, so
> it's not just this char.
> 
> Any clues one what is up?

I do note that in UnicodeData.txt we have

0587;ARMENIAN SMALL LIGATURE ECH YIWN;Ll;0;L;
  <compat> 0565 0582;;;;N;;;;;

for this case (and possibly the others I've had issues with): the char
does have a compatibility representation as a combination of others, and
it's those that show up with ConTeXt but not plain LuaTeX.
--
Joseph Wright