[luatex] Use cases for the token library

[Paul Isambert]

Patrick Gundlach <patrick at gundla.ch> a écrit:
> I keep asking myself what are use cases of the token library? The manual states:
> 
> "These functions are most useful when combined with the token_filter callback, ..."
> 
> but what are practical applications?

I gave an example in TUGboat paper (https://www.tug.org/members/TUGboat/tb32-1/tb100isambert.pdf),
but it wasn't very good (Ulrike Fischer incidentally gave a much better
solution in a message to this list; I say ``incidentally'' because she
wasn't discussing my paper at all).

Also, Taco faked XeTeX intercharstuff with the callback on the wiki
(http://wiki.luatex.org/index.php/Token_filter); the problem here of
course is that intercharstuff aren't very interesting in LuaTeX.

My position is that applications might be close to nonexistent, but it
would be a shame if the callback and functions didn't exist! As I see
it, the problem is that it's unlikely you always want something in the
callback, so if you're using it you're going to keep
registering/unregistering a function in it and it looks quite clumsy.

Perhaps an area where it may be useful, though, is to peek at the next
token(s) without setting their catcodes; I'm not sure there are many
applications for that, mind you.

Best,
Paul