[XeTeX] Future *TeX [was: XeTeX maintenance]

[Karljurgen Feuerherm]

Hello Simon,

Thanks very much for this detailed information. Obviously (for now) there
is plenty there that I don’t get, terminology-wise, but no surprise. I’ll
keep this handy and work on it as I can.

I never really worked with C (I was a B programmer back in the day) and am
now working on Swift (a good option for rewrite if there were to be one,
on OS X anyway… though of course that would make Windows-compatability
less good), but I can make do. Grew up on Pascal, though :)

Deeply appreciated. Getting more of us ‘mere’ users tech/(Xe)TeX-savvy is
vital to the future of the project, I think.

K

On 2015-04-26, 21:34, "Simon Cozens" <simon at simon-cozens.org> wrote:

>On 26/04/2015 23:02, Karljurgen Feuerherm wrote:
>> b) assuming a certain level of Xe(La)TeX competence at the Œpresenting¹
>> level, what recommendations would experts on this list make to Œupping
>>the
>> ante¹ I.e. progress toward a more insider understanding of the software?
>
>OK, I can answer this but I shall add a few other scattered thoughts as
>well. :-)
>
>First I also want to thank Khaled for his fantastic work in keeping
>xetex maintained, bugs fixed, and users happy. Until recently I presumed
>that Khaled was a professional software and/or typography person; I
>didn't realise that xetex was essentially a free time project for him. I
>was even more amazed by his dedication and professionalism.
>
>Bit by bit, I seem to have picked up how TeX and xetex works, and I wish
>I could help with the maintainership, but I just don't have the time at
>present. (And given that I think the future of TeX is spelt "SILE" I
>don't think it's appropriate for me to either. ;-) )
>
>How to get to know xetex? I think the first step in moving from a user
>to understanding the mechanics has to be Victor Eijkhout's TeX By Topic.
>Either buy a hard copy or download it from
>https://bitbucket.org/VictorEijkhout/tex-by-topic/src and read it over
>four or five times. It's by far the best introduction to how the TeX
>engine works.
>
>After that you should be able to work your way through the TeXBook; read
>that until you can understand the double-arrow sections.
>
>>From there, there are two directions you need to go in: the WEB program
>for TeX, and the xetex extensions and all the related font handling code
>and libraries that it uses.
>
>As Joseph and Phillip have mentioned, WEB is not an easy thing to work
>with, and WEB2C doesn't make it any better. But in a way there's nothing
>you can do about that; TeX is the WEB source. A lot of the design
>constraints of TeX in the early 80s don't apply any more; most of the
>unpleasantness around WEB comes from the fact that memory is allocated
>statically and that structures are hand-rolled with pointers and
>offsets. Rewriting the whole thing in another language wouldn't be a
>crazy idea (I've done it) and for long-term maintainability I think it's
>essential - we can't go on with statically-allocated PASCAL code for
>ever - but it would be a major operation outside of the bounds of
>maintaining the current *TeX projects.
>
>But the up side of that is that there's very little of the WEB code that
>you actually need to mess with. Most of it Just Works and is never going
>to need to change, and most of the time you can assume that if there's
>an problem, it's with the xetex-specific bits, rather than with Knuth's
>TeX.
>
>So after you have a conceptual understanding of how TeX works, the next
>step is to run weave on source/texk/web2c/xetexdir/xetex.web [1] and
>start reading. You can skim over parts 1-19, read the rest normally, and
>focus most of your attention on parts 37-46. In particular, you want to
>read over the parts which deal with native word nodes, which are
>(basically) hboxes containing native font characters. Look up
>"native_word_node" in the index at the back and read those sections.
>
>Many of the XeTeX extensions call out from Pascal into C; these are
>defined in the xetex.defines file. This is a bit tricky to match up
>because WEB2C (I think) strips the underscores from the names in the WEB
>file. So set_native_metrics in xetex.web gets turned into
>setnativemetrics, which is defined by xetex.h as measure_native_node,
>which you will find defined in XeTeX_ext.c - this is the key function
>which, takes a Pascal memory region representing a native word node (a
>bunch of Unicode characters), calls the font shaping functions on it,
>and fills in the height, width, and depth of that node back into Pascal
>so TeX can run its algorithms on it. Start your exploration of the C
>sources from that routine, and follow all the function calls until you
>understand what it's doing. At some point you will follow it down to the
>harfbuzz interface in XeTeXLayoutInterface.cpp and the FontConfig
>interface in XeTeXFontMgr_FC.cpp. (My feeling is that the
>AAT/Mac-specific stuff is dead now, and at any rate it's easier to
>understand FontConfig/harfbuzz anyway.)
>
>Once you get to that layer, you may be perplexed by the lack of
>documentation for both harfbuzz and FontConfig. Hopefully my article at
>http://www.simon-cozens.org/content/duffers-guide-fontconfig-and-harfbuzz
>might
>help with this.
>
>Finally, about the future of TeX. Obviously my view that a complete
>rewrite is a good idea is going to be a minority report for a while yet,
>so I'll stick to XeTeX and LuaTeX.
>
>I don't know as much as I should about LuaTeX. For me, the point of
>xetex is not just that it's a Unicode-compatible TeX, but also that it
>supports native fonts well and that it both handles native OS fonts in a
>simple way and supports shaping of complex scripts (the harfbuzz bit).
>Hans has stated that LuaTeX will not include external font shaping in
>the core, and Graham Douglas has done some experiments in calling out to
>harfbuzz in Lua for shaping, but I don't think it's a done thing yet.
>
>So if I want to use complex scripts and native fonts, (and I do) then
>right now xetex is where it's at for me.
>
>But in a way that's beside the point. I've come to thinking that the
>future of a software project has less to do with its features than with
>the size and dedication of its community. If xetex (or luatex) doesn't
>have a good core of people who can improve, debug and maintain it, then
>no matter how good it is, it isn't going to be the future of TeX.
>
>So I hope we can make that happen by building more understanding of the
>XeTeX internals, and I'm happy to mentor anyone who wants to understand
>the code better. If you have any questions about the code, ask on this
>list and I will try to answer.
>
>[1] Giving up on the change file idea was one of the smartest moves the
>xetex people made... Incidentally I attach weave-html.pl, which I
>thought was a standard TeX thing but turns out to be something I cooked
>up myself. It turns WEB files into pretty linked hypertext.