[luatex] LuaLaTeX's performance

[Pander]

On 2010-11-19 23:03, Khaled Hosny wrote:
> On Fri, Nov 19, 2010 at 09:23:46PM +0100, Pander wrote:
>> On 2010-11-19 21:15, Khaled Hosny wrote:
>>>> The other thing is the performance. Now it could be that the current
>>>> executable is not optimised and is doing loads of debug logging.
>>>> However, my font catalogue is a good test case.
>>>>
>>>> XeLaTeX can process it in two parts because other it runs out of memory.
>>>> However, LuaLaTeX is hitting some sort of performance barrier halfway my
>>>> document and after using up almost 6 GB of memory I have to kill off the
>>>> process. Please see attached screenshots of memory usage.
>>>
>>> LuaTeX will happily use all memory available to it (unlike most other
>>> TeX engines which have hard-coded memory limit, IIUC). The current font
>>> loading code is not very memory efficient and for certain fonts e.g.
>>> fonts with huge kerning tables (some font developers like to say their
>>> fonts have tens of thousands of kerning pairs, like if it is a feature
>>> or something). I think post-texlive2010 releases of luatex made some
>>> progress in this area, but I'm not sure if Hans started making use of it
>>> or not.
>>>
>>>> How can LuaLaTeX's performance be improved? Is the current executable in
>>>> TLContrib a debug build with maximum logging?
>>>
>>> I don't think so, I don't think it would be that relevant either. If you
>>> can isolate the most offending fonts it might help in identifying and
>>> possibly fixing the cause.
>>
>> It it difficult to isolate since gradually memory gets eaten. Can I run
>> it with profiling or logging to report which font is using how much memory?
> 
> Here is a quickly hacked script that takes a list of fonts and reports,
> in kb, the memory used for each:
> 
> for _,v in ipairs(arg) do
>     local b, m, f, t
>     b = string.explode(v, "/")
>     b = b[#b]
>     m = collectgarbage("count")
>     f = fontloader.open(v)
>     if f then
>         t = fontloader.to_table(f)
>         fontloader.close(f)
>         print(math.floor(collectgarbage("count") - m), b)
>         t = nil
>         collectgarbage('collect')
>     end
> end
> 
> Save it to some file and cal it something like:
> 
> $ texlua foo.lua `find /path/to/fonts`

There are some pretty big ones:

20208	CODE2001.ttf
37006	DoulosSILCR.ttf
37122	DoulosSILR.ttf

20479	DejaVuSerif-Bold.ttf
20479	DejaVuSerifCondensed-Bold.ttf
20480	DejaVuSerif-BoldItalic.ttf
20480	DejaVuSerifCondensed-BoldItalic.ttf
20550	DejaVuSerifCondensed.ttf
20550	DejaVuSerif.ttf
20558	DejaVuSerifCondensed-Italic.ttf
20558	DejaVuSerif-Italic.ttf
21014	DejaVuSansMono.ttf
21014	mplus-1m-bold.ttf
21014	mplus-1m-light.ttf
21014	mplus-1m-medium.ttf
21014	mplus-1mn-bold.ttf
21014	mplus-1mn-light.ttf
21014	mplus-1mn-medium.ttf
21014	mplus-1mn-regular.ttf
21014	mplus-1mn-thin.ttf
21014	mplus-1m-regular.ttf
21014	mplus-1m-thin.ttf
21014	mplus-2m-bold.ttf
21014	mplus-2m-light.ttf
21014	mplus-2m-medium.ttf
21014	mplus-2m-regular.ttf
21014	mplus-2m-thin.ttf
22583	DejaVuSans-BoldOblique.ttf
22583	DejaVuSansCondensed-BoldOblique.ttf
22638	DejaVuSans-Oblique.ttf
22639	DejaVuSansCondensed-Oblique.ttf
22657	mplus-1c-black.ttf
22657	mplus-1c-bold.ttf
22657	mplus-1c-heavy.ttf
22657	mplus-1c-light.ttf
22657	mplus-1c-medium.ttf
22657	mplus-1c-regular.ttf
22657	mplus-1c-thin.ttf
22657	mplus-1p-black.ttf
22657	mplus-1p-bold.ttf
22657	mplus-1p-heavy.ttf
22657	mplus-1p-light.ttf
22657	mplus-1p-medium.ttf
22657	mplus-1p-regular.ttf
22657	mplus-1p-thin.ttf
22687	mplus-2c-regular.ttf
22687	mplus-2p-black.ttf
22687	mplus-2p-bold.ttf
22687	mplus-2p-heavy.ttf
22687	mplus-2p-light.ttf
22687	mplus-2p-medium.ttf
22687	mplus-2p-regular.ttf
22687	mplus-2p-thin.ttf
22688	mplus-2c-black.ttf
22688	mplus-2c-bold.ttf
22688	mplus-2c-heavy.ttf
22688	mplus-2c-light.ttf
22688	mplus-2c-medium.ttf
22688	mplus-2c-thin.ttf
23970	DejaVuSans-Bold.ttf
24004	DejaVuSansCondensed-Bold.ttf
24175	DejaVuSansCondensed.ttf
24175	DejaVuSans.ttf
37110	CharisSILB.ttf
37110	CharisSILR.ttf
38445	CharisSILI.ttf
38447	CharisSILBI.ttf

18546	UMTypewriter-Regular.otf
18646	STIXGeneralItalic.otf
18689	STIXGeneralBolIta.otf
19196	STIXGeneralBol.otf
19266	cmunit.otf
19308	cmuntx.otf
19311	cmunbtl.otf
19311	cmunbto.otf
19316	cmuntb.otf
19481	OldStandard-Italic.otf
19520	OldStandard-Bold.otf
19569	OldStandard-Regular.otf
19675	cmunst.otf
19830	cmuntt.otf
20339	FreeSerifBoldItalic.otf
20628	STIXGeneral.otf
20771	FreeSerifBold.otf
21101	FreeSerifItalic.otf
21667	xits-math.otf
22912	cmunssdc.otf
22917	cmunso.otf
23083	cmunbso.otf
23087	cmunbmo.otf
23087	cmunbsr.otf
23088	cmunbmr.otf
23302	cmunbxo.otf
23303	cmunbbx.otf
23303	cmunsi.otf
23455	cmunsx.otf
23456	cmunss.otf
23614	cmunoti.otf
23615	cmunobi.otf
23643	FreeSans.otf
23665	cmunobx.otf
24009	cmunci.otf
24072	cmunui.otf
24076	cmunbi.otf
24118	cmunti.otf
24761	cmunbl.otf
24761	cmunsl.otf
24784	cmunorm.otf
24919	cmunbx.otf
25018	cmunrm.otf
28229	FreeSerif.otf

20341	FreeSerifBoldItalic.ttf
20772	FreeSerifBold.ttf
21070	FreeSerifItalic.ttf
23580	FreeSans.ttf
28262	FreeSerif.ttf

But can I run a profiler because the way memory is eaten and is not
being released is the limiting factor.

When I run XeLaTeX of LuaLaTeX on my font catalogue, the first one
reports to run out of memeory but the second just continues to eat up
all my real and virtual memory and eventually I have to kill it.

When I split my document in two parts, each about 135 fonts almost all
in regular, bold, bolditalic and italic, XeLaTeX is able to process it
and write about 10MB for each part. It takes my Ubuntu system to use 900
MB RAM and 200 MB swap to process one part (1/2). (Memory use increases
over time, does XeLaTeX use dynamic allocation?)

LuaLaTeX is unable to process one of these parts as it needs much much
more memory to do the job, so I have to kill it off again. When I split
it in 4 parts with each 65 fonts, it works. Each PDF is about 4 or 5 MB.
It takes my Ubuntu system to use 3500 MB RAM and 200 MB swap to process
this part (1/4). (That dynamic memory allocation occurs here is expected.)

In short, LuaLaTeX uses (for this font intensive document on this system
etc etc) approximately (2*(3500+200))/(900+200) = 6.7 times more space
in memory then XeLaTeX.

Regards,

Pander

> Regards,
>  Khaled
>