[XeTeX] problem with discretionary

[jfbu]

Hi,

regarding the character class issue 

(which isn't directly the one
I had reported at http://tug.org/pipermail/xetex/2017-March/027056.html
and again at top of this re-newed thread)

github user eg9 already did a report upstream

https://github.com/reutenauer/polyglossia/issues/145

as far as I can tell, the issue remains unresolved.

Best,

Jean-François

Le 3 déc. 2017 à 12:21, Zdenek Wagner <zdenek.wagner at gmail.com> a écrit :

> I do not know the exact revision of the change. The code which I sent tests the features, not the revision number.
> 
> Zdeněk Wagner
> http://ttsm.icpf.cas.cz/team/wagner.shtml
> http://icebearsoft.euweb.cz
> 
> 2017-12-03 12:18 GMT+01:00 jfbu <jfbu at free.fr>:
> 
> Only to point out frenchb.ldf (babel-french) does indeed
> 
>   \ifdim\the\XeTeXversion\XeTeXrevision pt<0.99994pt
>     \FB at nonchar=255 \relax
>   \else
>     \FB at nonchar=4095 \relax
>   \fi
> 
> whereas I see no similar thing in gloss-french.ldf
> 
> There seems to be two problems now, whereas
> I only had one initially
> 
> - my mwe does not compile with xetex 0.99992
> 
> - possibly, polyglossia-french has an issue with
> xetex 0.99994 and later
> 
> Jean-François
> 
> Le 3 déc. 2017 à 11:58, jfbu <jfbu at free.fr> a écrit :
> 
>> Thanks Zdeněk!
>> 
>> Should I thus conclude from this that polyglossia + French is currently broken ?
>> indeed the file gloss-french.ldf uses hardcoded 255 at various locations.
>> 
>> I am a bit lost though because my test mwe
>> 
>> \catcode`@ 11
>> \XeTeXinterchartokenstate=1
>> \newXeTeXintercharclass\french at punctthin 
>> \XeTeXcharclass `\; \french at punctthin
>>      \XeTeXinterchartoks 255 \french at punctthin = {\nobreak\thinspace}%
>> \catcode`;\active
>> \def;{\discretionary{\char`\;}{}{\char`\;}}
>> a;b
>> \bye
>> 
>> compiles fine with current XeTeX, but not with TL2015 XeTeX.
>> 
>> (the @ thing is only to stay close to control sequence names from gloss-french.ldf)
>> 
>> To clarify, the \def;{\discretionary{\char`\;}{}{\char`\;}} is analogous to
>> the kind of things Sphinx does in verbatim listings to allow linebreaks,
>> but isn't the exact thing.
>> 
>> Anyway, it does not originate from polyglossia nor
>> gloss-french.ldf but is a Sphinx add-on inside code listings.
>> 
>> If the problem can be solved by a patch at macro level, that would
>> be best, because it would allow the CPython internationalization
>> team to build their PDF docs without worrying about which XeTeX
>> they use, I notice some of their team uses Debian 2013.
>> 
>> Best
>> 
>> Jean-François
>> 
>> Le 3 déc. 2017 à 11:01, Zdenek Wagner <zdenek.wagner at gmail.com> a écrit :
>> 
>>> Hi,
>>> 
>>> please, notice that the number of character classes was increased from 256 to 4096, so 255 no longer works as a boundary but 4095 must be used. I use the following code that I took from some other package:
>>> 
>>> \edef\CSat{\the\catcode`\@} % in order to work in plain XeTeX
>>> \catcode`\@=11
>>> \ifdefined\e at alloc@intercharclass at top
>>>   \chardef\CSboundary=\e at alloc@intercharclass at top
>>> \else
>>>   \ifdefined\XeTeXinterwordspaceshaping
>>>     \chardef\CSboundary=4095 %
>>>     \def\newXeTeXintercharclass{%
>>>       \e at alloc\XeTeXcharclass\chardef
>>>               \xe at alloc@intercharclass\m at ne\@ucharclass at boundary}
>>>   \else
>>>     \chardef\CSboundary=255
>>>   \fi
>>> \fi
>>> \catcode`\@=\CSat
>>> 
>>> Afterwards I use \CSboundary instead of a fixed number. It thus works both with the old and new XeTeX.
>>> 
>>> 
>>> Zdeněk Wagner
>>> http://ttsm.icpf.cas.cz/team/wagner.shtml
>>> http://icebearsoft.euweb.cz
>>> 
>>> 2017-12-03 10:19 GMT+01:00 jfbu <jfbu at free.fr>:
>>> Hi,
>>> 
>>> I need some help to identify which XeTeX release fixed
>>> that problem, the mwe is
>>> 
>>> \catcode`@ 11
>>> \XeTeXinterchartokenstate=1
>>> \newXeTeXintercharclass\french at punctthin
>>> \XeTeXcharclass `\; \french at punctthin
>>>      \XeTeXinterchartoks 255 \french at punctthin = {\nobreak\thinspace}%
>>> \catcode`;\active
>>> \def;{\discretionary{\char`\;}{}{\char`\;}}
>>> a;b
>>> \bye
>>> 
>>> In  real life it appeared in a Polyglossia+French context
>>> with the semi-colon make active to insert a \discretionary
>>> similar to the above. There is no issue in lualatex.
>>> 
>>> It is currently seen at Python upstream (CPython) when
>>> they try to build French docs (via Sphinx)
>>> 
>>> https://bugs.python.org/issue31589
>>> 
>>> and it would be nice to pinpoint which XeTeX release
>>> precisely is ok. I know 0.99992 is bad and 0.99996 is good,
>>> but can't easily bisect.
>>> 
>>> Best,
>>> 
>>> Jean-François
>>> 
>>> 
>>> 
>>> 
>>> --------------------------------------------------
>>> Subscriptions, Archive, and List information, etc.:
>>>   http://tug.org/mailman/listinfo/xetex
>>> 
>> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tug.org/pipermail/xetex/attachments/20171203/b327f57b/attachment.html>