[luatex] LuaTeX versus ConTeXT Mk IV oddness

[Joseph Wright]

On 04/08/2014 18:45, Philipp Gesang wrote:
> ···<date: 2014-08-04, Monday>···<from: Joseph Wright>···
> 
>> Not sure which list to raise this one on, but hopefully there
>> will be enough people knowledgeable here to help!
>> 
>> Trying to read some UTF-8 data, I find that a few chars seem to
>> give unexpected results with ConTeXt MkIV comped to plain LuaTeX
>> using the same engine version:
[snip]

>> Any clues one what is up?
> 
> Context overloads the file reader. If you comment out the utf 
> related “textfileactions” in char-fio.lua (remake the format 
> afterwards) you get this:
> 
> ConTeXt  ver: 2014.07.30 10:31 MKIV beta  fmt: 2014.8.4  int:
> english/english system          > 'cont-new.mkiv' loaded open
> source     > 2 > 2 >
> /home/phg/context/tex/texmf-context/tex/context/base/cont-new.mkiv 
> close source    > 2 > 2 >
> /home/phg/context/tex/texmf-context/tex/context/base/cont-new.mkiv 
> system          > files > jobname 'textfileactions', input
> './textfileactions', result 'textfileactions' fonts           >
> latin modern fonts are not preloaded languages       > language
> 'en' is active open source     > 2 > 3 >
> /home/phg/src/context-dev/playground/textfileactions.tex
>> և:
> 
> tex error       > error on line 3 in file
> /home/phg/src/context-dev/playground/textfileactions.tex: ?
> 
> 
> \test #1#2\stop ^^@-\showtokens {#1:#2}
> 
> l.3 \test և\stop
> 
> 
> 1
> \ifdefined\normalshowtokens\let\showtokens\normalshowtokens\fi 2
> \def\test#1#2\stop{\showtokens{#1:#2}} 3 >>  \test և\stop 4

I'd come to the conclusion there had to be an overload involved
somewhere: thanks for the pointer.

> There was a thread a while ago about this interfering with Hebrew:
> http://www.ntg.nl/pipermail/ntg-context/2012/069168.html

That looks the 'other way' but I guess is fundamentally the same point
(of course, two chars to one doesn't mess up tokenization for me as I
never have two to start with).

> I haven’t checked for a while but back then it proved difficult to
> disable this preprocessing from within a document because they were
> being applied before the document was read.

I was wondering about that: whether there was a switch to turn it
on-and-off. Presumably that's not been included at present, so I'll
have to take a different tack with the code I have. (As you might
guess, my situation involves working with multiple engines so I can't
assume some chars get auto-converted.)
--
Joseph Wright