[XeTeX] (not) understanding XeTeXinterchartoks

[David Carlisle]

On 8 May 2015 at 13:44, Jonathan Kew <jfkthame at gmail.com> wrote:


> If what you want to do here doesn't involve inserting text,

yes, well what I wanted to do here was understand \XeTeXcharclass, so it doesn't
necessarily involve inserting text:-)

\XeTeXcharclass is a slightly strange beast as it is documented as
working at the level of
hlist character nodes, but it inserts tokens, and is specified in
terms of character tokens.
So I was just trying to understand the processing model, while
avoiding reading the code....

Hence the \futurelet which was giving me a hint as to where in the
pipeline I was, until I
looped.

> it appears that xetex sets the "boundary" state at the beginning of an interchartoks-inserted token list;

That is of course the key here, the rest of the behaviour is
understandable given that, so thanks!

I was partly lead down this path by a comment on an old example of
Taco's providing a similar feature
in luatex

http://tex.stackexchange.com/a/21691/1090

where the final comment is

@Taco: unfortunately, your code is not equivalent to what XeTeX does,
since your code inserts the tokens between each character-token in the
input stream, while XeTeX inserts them between character nodes being
added to the current list. ...


so I was lead to thinking about when exactly does xetex add these tokens....

David