[tex-k] A209 and A213: unexpandable active characters not considered

[胡亚捷 (Hu Yajie)]

(1) The bottom of page 209 of The TeXbook says

    Active characters have category 13, but you have to say `\noexpand
    <active character>' in order to suppress expansion when you are
    looking at such characters with \if or \ifcat.

This is true only for active characters defined as macros. If an active
character has been \let equal to a primitive command, the \noexpand is
unnecessary, and the category isn't 13:

--------------------------------------------------------------------------
*\catcode`\@=\active \let@=a

*\if at a\message{true}\fi          % character code = `a (instead of `@)
true
*\ifcat at a\message{true}\fi       % category code = 11 (instead of 13)
true
*\catcode`\@=\active \let@=\par

*\if@\vfill\message{true}\fi     % character code = 256 (instead of `@)
true
*\ifcat@\vfill\message{true}\fi  % category code = 16 (instead of 13)
true
--------------------------------------------------------------------------

This bug was discussed in 2017: https://tex.stackexchange.com/q/386480

(2) The \csname...\endcsname entry on page 213 of The TeXbook says that
only character tokens should remain after the tokens between \csname and
\endcsname are fully expanded. But the truth is that only non-active
character tokens should remain, and active characters will cause errors:

-----------------------------------------------------------
*\catcode`\@=\active \let@=a

*\csname foo at bar\endcsname
! Missing \endcsname inserted.
<to be read again>
                   @
<*> \csname foo@
                bar\endcsname
?
-----------------------------------------------------------

(3) The bottom of page 209 contains a deliberate lie:

    TeX will expand macros following \if [and \ifcat] until two unexpandable
    tokens are found. If either token is a control sequence, TeX considers it
    to have character code 256 and category code 16 [...]

The implementation in tex.web actually extends category codes to command codes
and considers control sequences to have command code 0 (\relax). The concept
of command codes isn't mentioned in The TeXbook, so Knuth supplied the
impossible category code 16 (although 0 also has a plausible interpretation).

This bug has been discovered long before and is mentioned in the bottom of
page 22 of the pTeX manual:

    TeXbook ni wa, orijinaru no TeX ni okeru \if to \ifcat no setsumei to
    shite

        If either token is a control sequence, TeX considers it to have
        character code 256 and category code 16, unless the current
        equivalent of that control sequence has been‌ \let equal to a
        non-active character token.
    
    to aru. Sunawachi

        \if ya \ifcat no hantei de wa (jissō no bengijō) kontorōru
        shīkuensu o kategorī kōdo 16 o motsu to minasareru
    
    to iu no de aru. Tokoro ga, tex.web no jissō wa kono tōri de naku,
    kontorōru shīkuensu o kategorī kōdo 0 to minashite iru. Sono tame,
    pTeX keiretsu ni oite wabun moji tōkun no \kcatcode no atai ga 16
    de aru baai mo, \ifcat hantei de kontorōru shīkuensu to kondō sareru
    koto wa nai.

Translation:

    In The TeXbook, the \if and \ifcat of the original TeX are explained as

        If either token is a control sequence, TeX considers it to have
        character code 256 and category code 16, unless the current
        equivalent of that control sequence has been‌ \let equal to a
        non-active character token.
    
    In other words,

        Control sequences are considered to have category code 16 in
        \if and \ifcat tests for the convenience of implementation.
    
    However, the implementation of tex.web is not as above, but rather
    considers control sequences to have category code 0. For that reason,
    Japanese character tokens with \kcatcode value 16 won't be confused
    with control sequences in \ifcat tests of the pTeX series.