[tex-eplain] \sidx not catcode-clean

[Dorai Sitaram]

Oleg K wrote:
> The problem is with the non-silent versions.  I don't think
> this is possible at all, because we need to typeset the
> arguments as non-verbatim, which is impossible to get from
> the verbatim version (except by writing to .aux and then
> reading, which sounds like a total mess...).

Alas, I think there's no choice if one wants "correctness".  User macros
can get away cutting these kinds of corners, but not something
general-purpose like Eplain.  

I would have just written the index argument to a temp file and immediately
read it back, instead of relying on the .aux file, where presumably the
"reading back" takes place during a subsequent run of TeX.  That can give
puzzling skew problems between edits, especially glaring (or, worse, _not_
glaring enough for the writer to suspect something's wrong) since the
nonsilent command prints its argument in the body text. 

By the way, what is the strike against writing to an external file and
reading it back (other than the fact that it's I/O and therefore has the
whiff of inefficiency)?  Unless there are lots of nonsilent index calls, we
should be able to suck it in, no?  (And the user could at their discretion
rewrite some or all of their nonsilents as silents to retrieve efficiency,
instead of putting up with strange behavioural differences between the two
types of index commands.)

Also, is there anything in eTeX (the TeX successor) that allows tokens to
be read again in different catcodes without diverting the text through an
external file?  If so, you could conditionalise the index code so that it
exploits eTeX... 

> One solution is just not do this for the non-silent
> versions, so that \verbatim only works with silent versions.
> However, then we have another problem -- since the two
> versions will treat their arguments differently, the same
> arguments given by the user will end up differently in the
> .idx file.  For example, compile this sidxtest.tex file:
>...
> 
> What I think can be done is to introduce yet another version
> of the indexing commands -- let's call them `verbatim'
> commands.  They will behave just like the silent commands,
> but their arguments will be read verbatim and will allow
> whatever crazy things inside them.  Since these commands
> will be new, no existing documents will be affected, and in
> the new documents the user will know what to expect, and
> will mix verbatim macros carefully with non-verbatim ones.
> 
> What do people think?

That should be OK.  You could even call it \index, and have it work exactly
like LaTeX's (i.e., no need to allow an [optional argument], which is just
cosmetic anyway, given the @ feature).  

Or you could have a flag that governs whether all subsequent \sidx (and
\idx) calls will be catcode-clean.  Since this is only an issue for those
users who index phrases containing funny characters, the onus of setting
the flag can be laid at their feet only.  For humanities-type use, running
the current versions of the commands without setting the flag would be
fine.

Dorai