[XeTeX] How to use intercharclasses (was "Issue with CJK in pdf build")

[Michiel Kamermans]

Scott Kohler wrote:
> As a "lurker" on this list, your explanation was very helpful to me. And yes, please, I'd like to read your explanation on how to use intercharclasses - seems like a feature you'd always want to use when typesetting multi-language documents.

Fair enough, I shall explain it to the best of my understanding, and 
Jonathan can jump in if I misrepresent the concept =)

The idea of intercharclasses is that characters can be assigned a class 
number, and that XeTeX will automatically insert specific TeX code 
between characters from one class, and characters from another class.

This behaviour can be turned on with the command:

\XeTeXinterchartokenstate = 1

Classes are numbered 0 through 255, but by default some characters are 
already assigned classes; all Latin characters are class 0 in XeTeX, all 
CJK characters are captured by classes 1, 2 and 3, class 254 has been 
tentatively reserved for a "wild card" class (ie, every class; this is 
useful when you have lots of uniform transition rules), and class 255 is 
used by the "boundary" characters (spaces and the like).

So how do we use it?

Say we have a document that mixes some ASCII text with CJK text, and we 
want to automatically switch fonts because the font that looks nice for 
ASCII doesn't support CJK, and the CJK font that we like has horribly 
disfigured and weirdly spaced ASCII characters. In order to make this 
happen, we need a few things: 1) fontspec, to conveniently load fonts, 
2) some font family definitions, so we can conveniently change fonts, 
and 3) transition rules for the different character classes, so that the 
right change command is issued at the right time. And we'll do all this 
in our preamble (although we could also do it in a separate 
package/style file):

1: \documentclass{article}
2: % turn on intercharclass behaviour
3: \XeTeXinterchartokenstate = 1
4: % require fontspec
5: \usepackage{fontspec}
6: % set up two fonts, one for latin, and one for CJK
7: \newfontfamily{\latinfont}{Times New Roman}
8: \newfontfamily{\cjkfont}{Ume Mincho}
9: % set up the transition rules - first from Latin (or boundary) to CJK
10: \XeTeXinterchartoks 0 1 = {\cjkfont}
11: \XeTeXinterchartoks 0 2 = {\cjkfont}
12: \XeTeXinterchartoks 0 3 = {\cjkfont}
13: \XeTeXinterchartoks 255 3 = {\cjkfont}
14: % then, from CJK (or boundary) to Latin
15: \XeTeXinterchartoks 1 0 = {\latinfont}
16: \XeTeXinterchartoks 2 0 = {\latinfont}
17: \XeTeXinterchartoks 3 0 = {\latinfont}
18: \XeTeXinterchartoks 255 0 = {\latinfont}
19: % finally, we need to be strict with Tex, since it should not resort 
to some unknown default font - force the issue:
20: \setmainfont{Times New Roman}
21: \begin{document}
22: ...our document text goes here. こんな風に。 Things should just work.
23: \end{document}

It is rather important not to forget lines 13, 18 and 20. If you do, a 
world of unpredictable results is yours.

This is the basic use, and it is already quite useful! But what if you 
want a language that isn't latin or CJK? The solution is to write your 
own "class definition", which means assigning characters to a specific 
class. For instance, say that in addition to Latin and CJK you also have 
a lot of box drawing to do, and for this you use the might convenient 
series of characters located between unicode points U+2500 and U+257F. 
We'll extend the previous bit of code to allow for this:

1: \documentclass{article}
2: % turn on intercharclass behaviour
3: \XeTeXinterchartokenstate = 1
4: % require fontspec
5: \usepackage{fontspec}
6: % set up three fonts, one for latin, one for CJK and one for box art
7: \newfontfamily{\latinfont}{Times New Roman}
8: \newfontfamily{\cjkfont}{Ume Mincho}
9: \newfontfamily{\boxfont}{Courier New}
10: % define a new character class for box drawing
11: \XeTeXcharclass `\┌ 4
12: \XeTeXcharclass `\┐ 4
13: \XeTeXcharclass `\└ 4
14: \XeTeXcharclass `\┘ 4
15: % ...
16: % extended transition rules... To latin:
17: \XeTeXinterchartoks 0 1 = {\cjkfont}
18: \XeTeXinterchartoks 0 2 = {\cjkfont}
19: \XeTeXinterchartoks 0 3 = {\cjkfont}
20: \XeTeXinterchartoks 4 3 = {\cjkfont}
21: \XeTeXinterchartoks 255 3 = {\cjkfont}
22: % To CJK:
23: \XeTeXinterchartoks 1 0 = {\latinfont}
24: \XeTeXinterchartoks 2 0 = {\latinfont}
25: \XeTeXinterchartoks 3 0 = {\latinfont}
26: \XeTeXinterchartoks 4 0 = {\latinfont}
27: \XeTeXinterchartoks 255 0 = {\latinfont}
28: % To box shapes:
29: \XeTeXinterchartoks 0 4 = {\boxfont}
30: \XeTeXinterchartoks 1 4 = {\boxfont}
31: \XeTeXinterchartoks 2 4 = {\boxfont}
32: \XeTeXinterchartoks 3 4 = {\boxfont}
33: \XeTeXinterchartoks 255 4 = {\boxfont}
34: % and force the main font to Times New Roman again
35: \setmainfont{Times New Roman}
36: \begin{document}
37: ...our document text goes here. こんな風に。 Things (┌┐) should just 
(└┘) work.
38: \end{document}

Note that we had to define a new font to use for our boxes, we had to 
say that all the characters that we use for box drawing are class 4, and 
we had to make sure to specify the transition rules, including making 
sure we add a "from class 4 to ..." to the latin and cjk rules (lines 20 
and 26)!

This gets dangerous the more classes you need, because magic numbers are 
bad (what does '4' mean, for instance? how do we know we can use it? 
what if some other package already used class 4, we'd be messing up that 
package's functionality!). We can rewrite the previous example to 
something a bit more useful, although this may not work on older 
versions of XeTeX because it uses a relatively new functionality: 
automatically getting the next free class number. For this we replace 
lines 10 through 15 with:

..: % define a new character class for box drawing, called boxclass
..: \newXeTeXintercharclass\boxclass
..: % build the entire block, from U+2500 to U+257F, using this new class
..: \XeTeXcharclass `\┌ \boxclass
..: \XeTeXcharclass `\┐ \boxclass
..: \XeTeXcharclass `\└ \boxclass
..: \XeTeXcharclass `\┘ \boxclass
..: \XeTeXcharclass `\... \boxclass
...etc

And lines 20, 26, and 29-33 will look like:

20: \XeTeXinterchartoks \boxclass 3 = {\cjkfont}
26: \XeTeXinterchartoks \boxclass 0 = {\latinfont}
29: \XeTeXinterchartoks 0 \boxclass = {\boxfont}
30: \XeTeXinterchartoks 1 \boxclass = {\boxfont}
31: \XeTeXinterchartoks 2 \boxclass = {\boxfont}
32: \XeTeXinterchartoks 3 \boxclass = {\boxfont}
33: \XeTeXinterchartoks 255 \boxclass = {\boxfont}

Already much better, rather than a magical number we now have something 
that clearly indicates what it's being used for, and we don't have to 
worry about what the underlying class number is. Things will just work.

Of course this is a hassle for class with lots of characters (good luck 
assigning all the CJK ideographs extension A characters to a new class 
character by character, for instance: there are 6582 of them!) so you 
may want to end up writing a command that runs through a for-loop of 
some kind to binds lots of characters to a class, rather than specifying 
them one by one, as well as a command that can deal with setting all the 
to-and-from transitions when you have lots of classes.

However, I won't bother giving examples of the code this would need, 
since I'm already writing a package that's supposed to do that, I'm just 
still in the process of refining it for general release (this package 
exploits the intercharclass system to bind fonts to unicode blocks, 
although Jonathan kindly pointed out that binding them to scripts 
instead of blocks is probably more sensible... which it is, I've just 
not had a great deal of time to make the switch) .

So hopefully this was useful as well, and you'll be going "oh, so it's 
actually pretty simple" now... basically as long as this explanation 
keeps you from going back to those Office applications just because you 
miss font switching, I'll feel like I made the world that tiny bit better =P

- Mike