[XeTeX] Confused (why are ignored characters not "removed from the input" as per Eijkhout's TeX by Topic ?)

[Jonathan Kew]

On 16/04/2023 17:16, Philip Taylor (RHBNC) wrote:
> Given —
> 
>> \catcode 9 = 9
>>
>> \let ~ = \undefined
>>
>> ^^I~
>>
>> \end
>>
> why does XeTeX report
> 
>> This is XeTeX, Version 3.141592653-2.6-0.999993 (TeX Live 2021/W32TeX) 
>> (preloaded format=xetex)
>>
>> restricted \write18 enabled.
>>
>> entering extended mode
>>
>> (./untitled-10.tex
>>
>> ! Undefined control sequence.
>>
>> l.3 ^^I~
>>
>> ?
>>
> This output makes it appear that the offending control sequence is (or 
> might be) ^^I~, whereas it is in fact simply ~.  Should not the ^^I have 
> been ignored rather than reported, as per line 1 ?  Note line 3 does not 
> really contain ^^I~ but rather <tab>~, but as tabs cannot be reliably 
> included in e-mail I represent them here as ^^I.  The ^^I in the 
> transcript is genuine.

First note: I see the same result with plain TeX. So not a XeTeX issue.

Anyway, this is expected behavior. The ^^I isn't part of the offending 
control sequence; it's just the preceding context on the line, which is 
what normally appears in a TeX error message.

Perhaps this is clearer if you add some more surrounding text:

   \catcode 9 = 9
   \let ~ = \undefined
   abc^^I~def
   \end

results in

   (./x.tex
   ! Undefined control sequence.
   l.3 abc^^I~
              def
   ?

Ignored characters are not "removed from the input" (despite anything 
Eijkhout says); they're still present, just ignored.

JK