[XeTeX] section styling breaks with sections, intercharclasses and fontspec

[Michiel Kamermans]

Hi all,

I'm running into a curious problem that I think is caused by an 
interplay by fontspec, intercharclass behaviour, and the way sections 
are dealt with by latex... so here goes.

I have a rather simple document,

\documentclass{article}
\usepackage{fontspec}
\newcommand{\cjkfont}{\setmainfont{Code2000}} % pick any font with 
Japanese support if you lack Code2000
\newcommand{\rmfont}{\setmainfont{Times New Roman}}
\rmfont
\XeTeXinterchartokenstate = 1
\XeTeXinterchartoks 0 1 = {\cjkfont}
\XeTeXinterchartoks 0 2 = {\cjkfont}
\XeTeXinterchartoks 0 3 = {\cjkfont}
\XeTeXinterchartoks 255 1 = {\cjkfont}
\XeTeXinterchartoks 255 2 = {\cjkfont}
\XeTeXinterchartoks 255 3 = {\cjkfont}
\XeTeXinterchartoks 1 0 = {\rmfont}
\XeTeXinterchartoks 2 0 = {\rmfont}
\XeTeXinterchartoks 3 0 = {\rmfont}
\XeTeXinterchartoks 1 255 = {\rmfont}
\XeTeXinterchartoks 2 255 = {\rmfont}
\XeTeXinterchartoks 3 255 = {\rmfont}
\begin{document}
\section{English, then 日本語, then English} % "English, then Japanese, 
then English"
Padding text for section
\section{日本語, then English} % Japanese, then English
Padding text for section
\end{document}

When running this through xelatex, the two section headings look 
different, and I don't know enough about (xe)(la)tex's internals to be 
able to say where the problem would live, and how to fix it. Hopefully 
someone on the list already encountered this problem before and has a 
solution for this problem.

To make it more interesting, if I replace the transition rules to just 
text insertion, a la:

\XeTeXinterchartoks 0 1 = { (e-j) }
\XeTeXinterchartoks 0 2 = { (e-j) }
\XeTeXinterchartoks 0 3 = { (e-j) }
\XeTeXinterchartoks 255 1 = { (e-j) }
\XeTeXinterchartoks 255 2 = { (e-j) }
\XeTeXinterchartoks 255 3 = { (e-j) }
\XeTeXinterchartoks 1 0 = { (j-e) }
\XeTeXinterchartoks 2 0 = { (j-e) }
\XeTeXinterchartoks 3 0 = { (j-e) }
\XeTeXinterchartoks 1 255 = { (j-e) }
\XeTeXinterchartoks 2 255 = { (j-e) }
\XeTeXinterchartoks 3 255 = { (j-e) }

Then this style breaking does not occur (although all the japanese will 
now look like "unknown glyph" characters).

Thanks in advance for any help on how to make the section (and of course 
chapter, subsection, subsubsection and paragraph) headings behave.

- Mike "Pomax" Kamermans
nihongoresources.com