[luatex] Using the token_filter callback only for 'some tokens'

[Joseph Wright]

Hello all,

I'm currently fiddling around with changing the case of (TeX) tokens in
Lua. Doing a UTF-8 case change on 'text' is easy enough once you get the
idea of using the unicode library it gives the 'wrong' result for
something like

  abc\foo

That can be avoided by doing a TeX loop to only pass on 'letters' to Lua
or by not passing on control words, but I'm looking to solve the problem
at the Lua end.

The idea of using the token_filter callback was suggested
(http://chat.stackexchange.com/transcript/message/13635878#13635878).
However, I don't see how to apply the callback to only the argument
rather than to 'all tokens'. For example, trying something like

\input luatexbase.sty %
\directlua{%
  local upper = unicode.utf8.upper
  local char = unicode.utf8.char
  mytokens = function()
    local t = token.get_next()
    if t[3] == 0 and t[1] == 11 then
      print(upper(char(t[2])), t[1], t[3])
    end
    return t
  end
}
\def\test#1{%
  \directlua{luatexbase.add_to_callback
    ("token_filter", mytokens,  "mytokens")}%
  #1%
  \directlua{luatexbase.remove_from_callback
    ("token_filter", "mytokens")}%
}

Before
\test{during}
after
\bye

Shows that as I guess I'd expect the filter is active until the end of
the line to remove it. In a real case that would clearly not work. I'm
doubtless missing something obvious here: what's the correct approach?
(I'm using luatexbase as this seems a little easier than the rather
callback.register, but if there is a solution with the latter I'll be
interested too.)
-- 
Joseph Wright