[luatex] LuaTeX versus ConTeXT Mk IV oddness

[Joseph Wright]

Hello all,

Not sure which list to raise this one on, but hopefully there will be
enough people knowledgeable here to help!

Trying to read some UTF-8 data, I find that a few chars seem to give
unexpected results with ConTeXt MkIV comped to plain LuaTeX using the
same engine version:

    \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi
    \def\test#1#2\stop{\showtokens{#1:#2}}
     \test և\stop

on my system gives

This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/W32TeX) (rev 4971)
 restricted \write18 enabled.
(./test.tex
> և:.
\test #1#2\stop ^^@-\showtokens {#1:#2}

l.3 \test և\stop

?

with "luatex" but

mtx-context     | warning: synctex is enabled
mtx-context     | run 1: luatex
--fmt="C:/texlive/2014/texmf-var/luatex-cache/context/a86c089b384a3076dc514ba966a1fac9/formats/luatex/cont-en"
--jobname="test"
--lua="C:/texlive/2014/texmf-var/luatex-cache/context/a86c089b384a3076dc514ba966a1fac9/formats/luatex/cont-en.lui"
--no-parse-first-line --synctex=1 --c:currentrun=1
--c:fulljobname="./test.tex" --c:input="./test.tex" --c:kindofrun=1
--c:maxnofruns=8 --c:synctex "cont-yes.mkiv"
This is LuaTeX, Version beta-0.79.1 (TeX Live 2014/W32TeX) (rev 4971)
 \write18 enabled.
open source     > 1 > 1 >
C:/texlive/2014/texmf-dist/tex/context/base/cont-yes.mkiv

ConTeXt  ver: 2014.05.21 22:04 MKIV beta  fmt: 2014.7.1  int:
english/english

system          > 'cont-new.mkiv' loaded
open source     > 2 > 2 >
C:/texlive/2014/texmf-dist/tex/context/base/cont-new.mkiv
close source    > 2 > 2 >
C:/texlive/2014/texmf-dist/tex/context/base/cont-new.mkiv
system          > files > jobname 'test', input './test', result 'test'
fonts           > latin modern fonts are not preloaded
languages       > language 'en' is active
open source     > 2 > 3 >
//ueahome3/stfsci2/c227/data/NTProfile/Desktop/test/test.tex
> ե:ւ

tex error       > error on line 3 in file
//ueahome3/stfsci2/c227/data/NTProfile/Desktop/test/test.tex: ?


\test #1#2\stop ^^@-\showtokens {#1:#2}

l.3 \test եւ\stop


1     \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi
2     \def\test#1#2\stop{\showtokens{#1:#2}}
3 >>  \test և\stop


?

using ConTeXT.

As you can see, the engines are the same and indeed I'm running exactly
the same file (encoded UTF-8). The issue here is not just that the
results are different but that I get a different number of tokens: in
the real situation that's a big issue. I've not tracked down all of the
codepoints this problem applies to, but there seem to be several, so
it's not just this char.

Any clues one what is up?
--
Joseph Wright