[tex-eplain] \sidx not catcode-clean

[Oleg Katsitadze]

On Wed, Apr 19, 2006 at 08:29:48AM -0400, Dorai Sitaram wrote:
> Allowing catcode changes within indexing commands is crucial, especially in
> computer-science documentation where many indexed items require verbatim. 

For the silent versions of the commands this is easy to
implement -- we just would need to read the arguments of all
\@sidx... commands verbatim, like you suggested for \href,
and then dump that into the .idx file.

The problem is with the non-silent versions.  I don't think
this is possible at all, because we need to typeset the
arguments as non-verbatim, which is impossible to get from
the verbatim version (except by writing to .aux and then
reading, which sounds like a total mess...).

One solution is just not do this for the non-silent
versions, so that \verbatim only works with silent versions.
However, then we have another problem -- since the two
versions will treat their arguments differently, the same
arguments given by the user will end up differently in the
.idx file.  For example, compile this sidxtest.tex file:

--------------------sidxtest.tex--------------------
\input eplain

\def\foo{foo}
\def\bar{bar}

\def\vidx{%
  \begingroup
    \uncatcodespecials
    \catcode`\{=1 \catcode`\}=2
    \finvidx
}
\def\finvidx#1{%
  \endgroup
  \sidx{#1}%
}%

\vidx{\foo\bar}

\sidx{\foo\bar}

\idx{\foo\bar}

\bye
--------------------sidxtest.tex--------------------

Makeindex thinks that the term from \vidx is different from
the other two, test.ind contains:

  \begin{theindex}

    \item \foo \bar , 1
    \item \foo\bar, 1

  \end{theindex}

This is because \@idxwrite uses \sanitize to convert the
(non-verbatim) argument into character tokens, and this puts
the spaces after the commands.  This will certainly break
many existing documents.

What I think can be done is to introduce yet another version
of the indexing commands -- let's call them `verbatim'
commands.  They will behave just like the silent commands,
but their arguments will be read verbatim and will allow
whatever crazy things inside them.  Since these commands
will be new, no existing documents will be affected, and in
the new documents the user will know what to expect, and
will mix verbatim macros carefully with non-verbatim ones.

What do people think?

Best,
Oleg