[XeTeX] Extended ^^ notation and \scantokens

[Joseph Wright]

Hello all,

Experimenting with \scantokens for generating characters from the
charcodes, the following issue comes up in XeTeX. For the test file

    \show ^^^^^10400
    \show ^^^^^^010400
    \def\gobble#1{}
    \showtokens\expandafter{%
      \romannumeral-`\q\expandafter\expandafter\expandafter\gobble
        \expandafter\string\csname
        \scantokens{^^^^^10400\noexpand}\endcsname%
    }
    \showtokens\expandafter{%
      \romannumeral-`\q\expandafter\expandafter\expandafter\gobble
        \expandafter\string\csname
        \scantokens{^^^^^^010400\noexpand}\endcsname%
    }

the \show statements work fine but the \scantokens versions don't. This
is not limited to the rather odd setup above (used so \showtokens is
applicable): with \everyeof{\noexpand} and a suitable set of \write
statements you see the same in a temporary file.

It seems that something is up once you get to five hexadecimal digits:
bug in XeTeX?
--
Joseph Wright