[XeTeX] problem using interchartoks

[Jonathan Kew]

On 8 Jan 2009, at 02:54, Michiel Kamermans wrote:

> Hi, I'm trying to get the following bit of xetex to work:
>
> \documentclass[12pt, openright, a4paper]{book}
>
> \usepackage{fontspec}
>
> \setmainfont{Palatino Linotype}
>
> % set up inter-charclass behaviour
>
> \XeTeXinterchartokenstate = 1
>
> % when going from not CJK to CJK
>
> \XeTeXinterchartoks 0 1 = { TO_CJK1 }
>
> \XeTeXinterchartoks 0 2 = { TO_CJK2 }
>
> \XeTeXinterchartoks 0 3 = { TO_CJK3 }
>
> % when going from CJK to not CJK
>
> \XeTeXinterchartoks 1 0 = { 1TO_REST }
>
> \XeTeXinterchartoks 2 0 = { 2TO_REST }
>
> \XeTeXinterchartoks 3 0 = { 3TO_REST }
>
> \begin{document}
>
> This is a test to see whether or not にほんご or 日本語 is  
> properly handled.
>
> \end{document}

Close, but not quite correct. Spaces (glue) and other non-character  
stuff is treated as the special class value 255, so because you've got  
spaces around your CJK text, the transitions will be from/to class  
255, not class 0. The latter would apply if you had no space between  
the Latin and CJK chars.

So if you change your \XeTeXinterchartoks declarations to use 255  
instead of 0, you'll see the results you expected. (In practice, you'd  
probably want to handle both 0 and 255.)

(Note that you'll also need to change the underscores in your test to  
something else (e.g., hyphens), otherwise you'll get an error about  
missing math-mode switches when xetex encounters them.)

You might like to look at the tools provided by the zhspacing package  
and/or xeCJK, which build on these primitives to provide richer  
support for Chinese.

HTH,   JK