[luatex] own tokenizer
wolfgang at cs.ioc.ee
Thu Aug 1 15:09:37 CEST 2013
according to my understanding of the LuaTeX reference, there are two
callbacks that you can use to influence the way, input is turned into
• The process_input_buffer callback turns plain input lines into plain
• The token_filter callback essentially turns token lists into token
However, there seems to be no direct way of providing your own
translator from input lines to token lists; the actual tokenization
seems to be always done by LuaTeX on its own.
I have thought about a solution to this problem: I would set the
catcodes of the characters \, ^^M, ^^@, %, and ^^? to “other”. This way,
LuaTeX’s tokenizer would generate a token from each single character.
Then I would implement a token_filter callback that fetches input
characters by fetching tokens.
Is this approach sensible? Are there better ways of achieving what I
More information about the luatex