[lltx] pattern loading in luatex

[Manuel Pégourié-Gonnard]

Le 05/03/2011 18:22, Stephan Hennig a écrit :
> AFAIK, LuaTeX uses a more complex trie with pre-calculated fall-back
> states.

I wasn't aware of that, good to know.

> Are there any numbers about what makes loading and processing
> (many) patterns in LuaTeX slow?  Is it the first stage of building the
> traditional pattern trie from text files or is it the calculation of
> fall-back states afterwards?
> 
Well, I just ran some tests, all with the Plain + etex.src format. First, I had
pdfTeX generate this format with USenglish only and with all languages
available. Next, I tried loading a format with LuaTeX from TeX Live 2009, either
with only USenglish or with all languages. Finally, I had LuaTeX 0.65 with
formats from TL 2010 process a file loading only USenglish or all languages.
Here are the results:

pdftex (TL 2010) fmt gen, USenglish only:       0.3s
pdftex (TL 2010) fmt gen, all lang:             3.3s

luatex (TL 2009) fmt load, USen only:           0.4s
luatex (TL 2009) fmt load, all lang:            6.1s

luatex 0.65 (TL 2010) \uselanguage{USenglish}:  0.5s
luatex 0.65 (TL 2010) \uselanguage{<ALL>}:      6.8s

So, it looks like processing all languages at format generation time takes some
3 seconds to pdfTeX. Processing them at format loading time takes some 5.7
seconds to LuaTeX, while reading from external files and processing them takes
something like 6.3 seconds.

So, unless I missed something, it looks like pattern processing with LuaTeX
takes almost twice the times is takes with pdfTeX. Not sure if it is related to
the generation of the fall-back states you mention or some other change in the
internal code.

Anyway, you rarely need all 85 languages in one single document, so probably
it's not a big deal in normal situations.

>> Option 2 is implemented since TL 2010.
> 
> Great!  Somehow I happened to followed that back then, but missed what
> it was all about. :/
> 
IIRC, the discussions weren't exactly easy to follow, and there was already
enough questions about the multiple versions of German patterns to keep you busy :-)

Manuel.