[luatex] callbacks in luatex 0.89

[Hans Hagen]

On 2/13/2016 3:21 AM, Will Robertson wrote:
> On 12 Feb 2016, at 10:08 PM, Hans Hagen <pragma at wxs.nl> wrote:
>>
>> injecting stuff (at the user level) in the input stream in an uncontrolled way is messy anyway but if one really wants to do that, one can easily kick in a callback when reading the file and then add commands or whatever at unmarked script/language boundaries (after all it's mostly trying to make structure from non structure we're talking of)
>
> Working directly before the input stream is processed seems unlikely to be able address things like
>
>      \def\x{x}
>      \def\y{<chinese>}
>      \x\y

indeed, and a token injector has the same issue as it takes action at 
some spot that might suits some use but not another use so it's always 
half a solution (which is why we've chosen for callbacks at several spots)

> And using virtual fonts to add Chinese to a roman font is not enough because you also have to switch the language setup. Oh I see Ulrike already said that :)

sure but injecting tokens into the stream based on character properties 
is just as not-enough as it can badly interfere with whatever gets 
injected etc etc (i bet grouping is a nightmare) so i would never use 
such a method in context

> Since XeTeX’s interchartoken feature will never be provided directly in LuaTeX (I assume), it would be nice at least if it could be emulated by 3rd party authors so that XeTeX and LuaTeX code can be shared. So I’d support this proposal:

you assume right (and compatibility with other engines is not on the 
agenda as one can just use the other engines)

>> we might add a token.print function (if so it will be around version 0.95) in which case one can use token.get_tok in an endless loop to pick up tokens and print them back but (as with the callback) it will be very inefficient
>
> … even if it is inefficient :)

i experimeted with it and it will probably show up (but one probably has 
to fight a bit with expansion, lookahead and input levels)

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
       tel: 038 477 53 69 | www.pragma-ade.com | www.pragma-pod.nl
-----------------------------------------------------------------