[luatex] [lltx] Speeding up otf lua cache file load

[Reinhard Kotucha]

On 2013-03-28 at 10:09:18 +0100, Paul Isambert wrote:

 > Reinhard Kotucha <reinhard.kotucha at web.de> a écrit:
 > > I copied this thread from [lltx] to [luatex] because I fear that the
 > > luaotfload maintainer is not listening on [lltx]
 > >
 > >
 > > On 2013-03-23 at 17:59:23 +0400, LS wrote:
 > >
 > >  >  Hi, all,
 > >
 > >  > I think, that compiling  with otf font would be very quick if
 > >  > it would be possible  to create a separate
 > >  > "temp-<font-name>.lua" in a current directory only with the
 > >  > project used glyphs.  It could be a package
 > >  > f.e. "glyphstoluahash.sty" witch would do all that. Maybe, on
 > >  > the first run it would be slow, while it will pick all these
 > >  > glyphs, but on other runs, if  "temp-<font-name>.lua" exists
 > >  > in a current directory, it would do nothing, but
 > >  > luaotfload > would use this local hash file.
 > >  > How many different glyphs uses articles, probably less than
 > >  > 500 from one font. So it would be a huge improvement.
 > >  >
 > >  > I see in a otfl-luat-dum.lua that $TEXMFCACHE can be set for
 > >  > the cache look up, but when I set it to:
 > >  > TEXMFCACHE=.//;$TEXMFVAR luatex crates a local empty
 > >  > directories ./luatex-cache/generic. I tried to copy font cache
 > >  > file in local directory and in a created one, but it still
 > >  > loads cache files from 
 > >  > $TEXMFVAR\luatex-cache\generic\fonts\otf\
 > >  >
 > >  > So, what do you think about that idea?
 > >
 > > A better approach is to pre-compile the Lua files.  A
 > > pre-compiled Lua file is loaded significantly faster because it
 > > doesn't have to be parsed.  I observed that loading a .luc file
 > > is up to 30 times as fast as loading .lua file.
 > >
 > > You can try yourself.  Since Lua recognizes whether a file is
 > > pre-compiled, regardless of the extension, you can do
 > >
 > >   texluac -s -o foo.luc foo.lua
 > >   move foo.luc foo.lua
 > >
 > > for testing.
 > >
 > > What I nowadys do in my own programs in order to load large files with
 > > dofile() is to provide a wrapper, let's call it do_file().  The
 > > wrapper takes a file name as an argument.  If the file name has an
 > > extension, then the argument is passed directly to dofile().
 > > If no extension is given, the wrapper looks for foo.luc and foo.lua.
 > > If both are present, the newer one is used.
 > >
 > > I think that this approach can be used by luaotfload too without much
 > > effort.  The filename cache can be large, but the Lua representation
 > > of OTFs can be huge.  Pre-compiling the Lua files helps in both cases.
 > >
 > > Vafa, AFAIR you offered to maintain luaotfload some time ago.  What do
 > > you think about this suggestion?  See the function do_file() below.
 > >
 > > BTW, luaotfload creates files containing absolute paths.  In my
 > > current application it's a pain.  Thus the function below is using
 > > kpathsea in order to locate files.  It's the preferred way to locate
 > > files in TeX Live anyway.
 > 
 > I've never really tried to make font loading faster, but in my
 > experience the bulk of the time spent processing comes from setting
 > kern tables to the demanded size. Loading Minion Pro twenty times
 > with:
 > 
 >   \directlua{T = os.time()}
 >   % Loading the font 20 times.
 >   \directlua{texio.write_nl("TIME: " .. os.time()-T)}
 > 
 > returns "2" if kern tables are set, "0" otherwise. Using pre-compiled
 > Lua files doesn't make that faster (the loading of the cache file is
 > negligible compared to its processing). A font that is lighter on kern
 > tables (e.g. Palatino Linotype) also loads much faster.

Linas tried the function I attached to my previous mail and sent me
the result:

 | Here's what I've got testing one font:
 |
 | Lua size: 13MB
 | Luc size: ~8MB.
 |
 | Time loading lua: ~5.8s.
 | Time loading luc: ~1.5s.

He only loaded the file and didn't process it further.
However, I can't reproduce this at all.  Pre-compiling *all* cached
fonts takes 2.4s normally and 3.6s with an empty filesystem cache.

Loading:

  temp-charissilr.lua  2.4MB  160ms
  temp-charissilr.luc  1.5MB   45ms

Thus, on my system there will be only a noticable difference if many
large fonts are used.  However, I can't neglect Linas' experience,
though I have no idea why it's *that* slow there.
 
 > One solution I've considered but never implemented is to cache fonts
 > at a given size, the one most used by the user (e.g. 10pt); then it
 > would be faster by default, unless loading at a different size (the
 > font could also be cached with several sizes too).

The program I wrote recently had to be optimized for speed because
it's run in the background and people expect a reaction immediately
after pressing a button.  The solution was to load luaotfload and
declare the fonts when the format file is created.  I only need two
fonts (CharisSIL, Regular and Bold) at three different sizes.  Thus,
this approach is possible.  I currently avoid fontspec.  It blows up
the format file significantly and slows down everything.

When the format file is loaded, os.clock() returns 0.4s.  If I remember
correctly, whithout the preloaded fonts it returned 0.25s.  The
CharisSIL fonts and their Lua representations are quite large, much
larger than Latin Modern.  So I'm quite satisfied with the result.
Generating a PDF file takes 0.6 or 0.7 seconds, depending on its
content.  The former version (fontspec and without a dedicated format
file) took at least 1.6s.  I can't determine the total time under
Windows, but it's slower there, especially if it's run the first time.

There is not much text in the PDF files, just a few small tables.
Maybe this is the reason why the font subsetting doesn't take much
time.  The main content of the file are ECG plots created with
\pdfliteral.  BTW, my first attempt was to use Metapost but it was
much too slow.  Processing data with Lua and writing PDF stuff
directly into the PDF file is extremely fast.  It's similar to what
you described in a TUGboat article.  The difference is that your
approach is more Metapost orientated while mine is more PostScript
orientated, just because I don't need a user interface.

 > Of course, reducing cache files to a subset of the glyphs would
 > also make things faster, provided only the relevant parts of the
 > kern table are processed. How much faster it would be, I don't
 > know, though.

What would this subset contain?  The glyphs you're currently using in
a particular document or all the glyphs needed for a particular
language?

Regards,
  Reinhard

-- 
----------------------------------------------------------------------------
Reinhard Kotucha                                      Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover                              mailto:reinhard.kotucha at web.de
----------------------------------------------------------------------------
Microsoft isn't the answer. Microsoft is the question, and the answer is NO.
----------------------------------------------------------------------------