[XeTeX] Extended ^^ notation and \scantokens

[Qing Lee]

On 2014-05-05 at 17:25:08 +0800 GMT, Joseph Wright wrote:
> Hello all,
>
> Experimenting with \scantokens for generating characters from the
> charcodes, the following issue comes up in XeTeX. For the test file
>
>      \show ^^^^^10400
>      \show ^^^^^^010400
>      \def\gobble#1{}
>      \showtokens\expandafter{%
>        \romannumeral-`\q\expandafter\expandafter\expandafter\gobble
>          \expandafter\string\csname
>          \scantokens{^^^^^10400\noexpand}\endcsname%
>      }
>      \showtokens\expandafter{%
>        \romannumeral-`\q\expandafter\expandafter\expandafter\gobble
>          \expandafter\string\csname
>          \scantokens{^^^^^^010400\noexpand}\endcsname%
>      }
>
> the \show statements work fine but the \scantokens versions don't. This
> is not limited to the rather odd setup above (used so \showtokens is
> applicable): with \everyeof{\noexpand} and a suitable set of \write
> statements you see the same in a temporary file.
>
> It seems that something is up once you get to five hexadecimal digits:
> bug in XeTeX?
> --
> Joseph Wright
>
>
>
> --------------------------------------------------
> Subscriptions, Archive, and List information, etc.:
>    http://tug.org/mailman/listinfo/xetex
>

It seems to be related to the following tickets:

http://sourceforge.net/p/xetex/bugs/79/
http://sourceforge.net/p/xetex/bugs/80/
http://sourceforge.net/p/xetex/bugs/88/

Qing Lee