[XeTeX] problem with discretionary

[jfbu]

Sorry for noise as

https://github.com/reutenauer/polyglossia/pull/178

has been merged upstream.

The question now is when will this make its way to CTAN or TeXLive.

As far as I can tell CTAN has 1.42.4 and the above was merged into 1.42.5.

(this is secondary topic to this thread, though)

Best,

Jean-François

Le 3 déc. 2017 à 15:34, jfbu <jfbu at free.fr> a écrit :

> 
> Hi,
> 
> regarding the character class issue 
> 
> (which isn't directly the one
> I had reported at http://tug.org/pipermail/xetex/2017-March/027056.html
> and again at top of this re-newed thread)
> 
> github user eg9 already did a report upstream
> 
> https://github.com/reutenauer/polyglossia/issues/145
> 
> as far as I can tell, the issue remains unresolved.
> 
> Best,
> 
> Jean-François
> 
> Le 3 déc. 2017 à 12:21, Zdenek Wagner <zdenek.wagner at gmail.com> a écrit :
> 
>> I do not know the exact revision of the change. The code which I sent tests the features, not the revision number.
>> 
>> Zdeněk Wagner
>> http://ttsm.icpf.cas.cz/team/wagner.shtml
>> http://icebearsoft.euweb.cz
>> 
>> 2017-12-03 12:18 GMT+01:00 jfbu <jfbu at free.fr>:
>> 
>> Only to point out frenchb.ldf (babel-french) does indeed
>> 
>>   \ifdim\the\XeTeXversion\XeTeXrevision pt<0.99994pt
>>     \FB at nonchar=255 \relax
>>   \else
>>     \FB at nonchar=4095 \relax
>>   \fi
>> 
>> whereas I see no similar thing in gloss-french.ldf
>> 
>> There seems to be two problems now, whereas
>> I only had one initially
>> 
>> - my mwe does not compile with xetex 0.99992
>> 
>> - possibly, polyglossia-french has an issue with
>> xetex 0.99994 and later
>> 
>> Jean-François
>> 
>> Le 3 déc. 2017 à 11:58, jfbu <jfbu at free.fr> a écrit :
>> 
>>> Thanks Zdeněk!
>>> 
>>> Should I thus conclude from this that polyglossia + French is currently broken ?
>>> indeed the file gloss-french.ldf uses hardcoded 255 at various locations.
>>> 
>>> I am a bit lost though because my test mwe
>>> 
>>> \catcode`@ 11
>>> \XeTeXinterchartokenstate=1
>>> \newXeTeXintercharclass\french at punctthin 
>>> \XeTeXcharclass `\; \french at punctthin
>>>      \XeTeXinterchartoks 255 \french at punctthin = {\nobreak\thinspace}%
>>> \catcode`;\active
>>> \def;{\discretionary{\char`\;}{}{\char`\;}}
>>> a;b
>>> \bye
>>> 
>>> compiles fine with current XeTeX, but not with TL2015 XeTeX.
>>> 
>>> (the @ thing is only to stay close to control sequence names from gloss-french.ldf)
>>> 
>>> To clarify, the \def;{\discretionary{\char`\;}{}{\char`\;}} is analogous to
>>> the kind of things Sphinx does in verbatim listings to allow linebreaks,
>>> but isn't the exact thing.
>>> 
>>> Anyway, it does not originate from polyglossia nor
>>> gloss-french.ldf but is a Sphinx add-on inside code listings.
>>> 
>>> If the problem can be solved by a patch at macro level, that would
>>> be best, because it would allow the CPython internationalization
>>> team to build their PDF docs without worrying about which XeTeX
>>> they use, I notice some of their team uses Debian 2013.
>>> 
>>> Best
>>> 
>>> Jean-François
>>> 
>>> Le 3 déc. 2017 à 11:01, Zdenek Wagner <zdenek.wagner at gmail.com> a écrit :
>>> 
>>>> Hi,
>>>> 
>>>> please, notice that the number of character classes was increased from 256 to 4096, so 255 no longer works as a boundary but 4095 must be used. I use the following code that I took from some other package:
>>>> 
>>>> \edef\CSat{\the\catcode`\@} % in order to work in plain XeTeX
>>>> \catcode`\@=11
>>>> \ifdefined\e at alloc@intercharclass at top
>>>>   \chardef\CSboundary=\e at alloc@intercharclass at top
>>>> \else
>>>>   \ifdefined\XeTeXinterwordspaceshaping
>>>>     \chardef\CSboundary=4095 %
>>>>     \def\newXeTeXintercharclass{%
>>>>       \e at alloc\XeTeXcharclass\chardef
>>>>               \xe at alloc@intercharclass\m at ne\@ucharclass at boundary}
>>>>   \else
>>>>     \chardef\CSboundary=255
>>>>   \fi
>>>> \fi
>>>> \catcode`\@=\CSat
>>>> 
>>>> Afterwards I use \CSboundary instead of a fixed number. It thus works both with the old and new XeTeX.
>>>> 
>>>> 
>>>> Zdeněk Wagner
>>>> http://ttsm.icpf.cas.cz/team/wagner.shtml
>>>> http://icebearsoft.euweb.cz
>>>> 
>>>> 2017-12-03 10:19 GMT+01:00 jfbu <jfbu at free.fr>:
>>>> Hi,
>>>> 
>>>> I need some help to identify which XeTeX release fixed
>>>> that problem, the mwe is
>>>> 
>>>> \catcode`@ 11
>>>> \XeTeXinterchartokenstate=1
>>>> \newXeTeXintercharclass\french at punctthin
>>>> \XeTeXcharclass `\; \french at punctthin
>>>>      \XeTeXinterchartoks 255 \french at punctthin = {\nobreak\thinspace}%
>>>> \catcode`;\active
>>>> \def;{\discretionary{\char`\;}{}{\char`\;}}
>>>> a;b
>>>> \bye
>>>> 
>>>> In  real life it appeared in a Polyglossia+French context
>>>> with the semi-colon make active to insert a \discretionary
>>>> similar to the above. There is no issue in lualatex.
>>>> 
>>>> It is currently seen at Python upstream (CPython) when
>>>> they try to build French docs (via Sphinx)
>>>> 
>>>> https://bugs.python.org/issue31589
>>>> 
>>>> and it would be nice to pinpoint which XeTeX release
>>>> precisely is ok. I know 0.99992 is bad and 0.99996 is good,
>>>> but can't easily bisect.
>>>> 
>>>> Best,
>>>> 
>>>> Jean-François
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --------------------------------------------------
>>>> Subscriptions, Archive, and List information, etc.:
>>>>   http://tug.org/mailman/listinfo/xetex
>>>> 
>>> 
>> 
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tug.org/pipermail/xetex/attachments/20171203/930cce0f/attachment-0001.html>