[luatex] \luaescapestring and \escapechar

[Heiko Oberdiek]

On 16.05.2014 22:41, Joseph Wright wrote:
> Hello all,
> 
> Running some tests today I stumbled across the following issue. With the
> demo:
> 
>     \def\test{\luaescapestring{\noexpand\foo}}
>     \escapechar=-1\relax
>     \test
> 
> I get
> 
>     This is LuaTeX, Version beta-0.79.1 (TeX Live 2014) (rev 4971)
>      restricted \write18 enabled.
>     (./test.tex
>     ! String contains an invalid utf-8 sequence.
>     test ^^@-luaescapestring {noexpand foo }
> 
>     l.3 \test
> 
>     ?
> 
> I was as you'd imagined a bit surprised: I spent a while looking over my
> source for dodgy input before concluding there is something more subtle up.
> 
> Is this in some way deliberate or a bug?

Clearly a bug.

TeX behavior: If \escapechar is negative or greater than 255 the output
of the escape char is suppressed (via \string, writing unexpandable
commands to files, ...).

LuaTeX: Of course the behavior is different for values greater than 255,
because LuaTeX supports Unicode characters. But negative values are erratic:

    \def\test#1{%
      \escapechar=#1\relax
      \immediate\write16{<\relax>}%
    }
    \test{65}% A
    \test{-1}
    \test{-2}
    \test{-127}
    \test{-128}
    \test{-129}
    \test{-254}
    \test{-255}
    \test{-256}
    \test{-257}
    \test{-959}
    \csname @@end\endcsname\end

Result for "tex":
    65: "A"
    0:  ^^@
    other: no escape char

Result for "luatex":

    65: "A"
    0: nothing, but also the rest of the line "relax >" is gone!!
    -1: 0xFF
    -2: 0xFE
    -127: 0x81
    -128: 0x80
    -129: 0x7F
    -254: 0x02
    -255: 0x01
    -256: nothing, but also the rest of the line "relax >" is gone!!
    -257: 0xFF
    -959: "A"

=> Two bugs:
   * \escapechar=0, then the output of the escape char should
      by character zero (^^@) instead of an unexpected end of string
      that kills characters behind.
   * Negative values of \escapechar: The escape char should not
     be printed at all.

Yours sincerely
  Heiko Oberdiek