[luatex] token_filter

[Élie Roux]

Dear all,

> I second that; going for token_filter will only result in hacks, I’m afraid.
> It’s just so easy with nodes. I think Taco’s page on the wiki was just for fun.

I'll do that then. I think the best is even to hook in the luaotfload 
code, and inspire from what Hans did in ConTeXt.

> Elie’s original questions:
>>    - where is the ouptut of token.command_name documented?
>
> Err, in the manual?

Section 4.17.6 reads:

"
This returns the name associated with the ‘command’ value of the token 
in LUATEX. There is not always a direct connection between these names 
and primitives. For instance, all \ifxxx tests are grouped under 
if_test, and the ‘command modifier’ defines which test is to be run.
"

This is good, but where do I get the different possible outputs, and 
what they correspond to? Or is there something I missed in the manual?

>>    - how can I know if the token I get is inside a csname? (ex: in a file
>> I have \foobar. When, in the token_filter, I get the token b, how can I
>> know it's part of a csname name ?)
>
> “b” in “\foobar” isn’t a token; the token is the entire command.

Well, that's what I thought at the beginning, but no... if I make a dumb 
print function in the callback in a very simple plain tex file it's 
true, but when it starts to be in a complex LaTeX document, I get *a 
lot* of things coming from csnames, put some prints in my version of 
polyglossia and you'll see very surprising things! Maybe there is 
something I did wrong, but it's quite hard to figure out...

>>    - same question for other characters, for example '<' in \ifnum\foo<0
>
> If I understand your question correctly as “How can I know whether ‘<’
> is used as an operator in an \ifnum-conditional”, then the answer is:
> keep track of what has come before :) (Or, actually, perhaps some
> tracing is possible in Lua: “tex.tracingifs”... “tex.currentconditional”...?)

Ok.

> Somebody (Patrick, if I remember correctly) once asked on this list
> what token_filter was good for; honestly, I don’t know. It’s nice to
> have it around when you really don’t know how to spend/waste a few
> hours at your computer, but apart from that, I find it neither useful
> nor usable.

That's what I understood after several hours :)

Thank you,
-- 
Elie