# [luatex] [lltx] Speeding up otf lua cache file load

Paul Isambert zappathustra at free.fr
Sat Mar 30 23:28:58 CET 2013

Arno Trautmann <Arno.Trautmann at gmx.de> a écrit:
> Hi all,
>
> Paul Isambert wrote:
> > Reinhard Kotucha <reinhard.kotucha at web.de> a écrit:
> >> On 2013-03-29 at 23:35:54 +0100, Paul Isambert wrote:
> >>
> >>   > Dumping formats in order to save time is probably a very good
> >>   > solution, even though not very widespread as far as I can tell (I
> >>   > myself don't use it, but I'm constantly modifying the files I use...).
> >>
> >> I could do it because users can't change things anyway.  In medical
> >> environments they aren't even allowed to install software themselves
> >> and the IT departments will not change anything because they don't
> >> want to be responsible if somethings goes wrong.
> >>
> >> In general it's quite problematic to provide dedicated format files.
> >> Suppose you've written a LaTeX .cls file for your students in order to
> >> write their theses.  Providing a dedicated format file would speed up
> >> things significantly, especially if packages like tikz or pgfplots are
> >> used, which load zillions of files.
> >>
> >> But format files are not portable among different versions of a
> >> particular program, a format file created under TeX Live 2012 will not
> >> work under TeX Live 2013.  The only reasonable way to distribute such
> >> a .cls file is to provide a .zip file which can be extracted either in
> >> TEXMFLOCAL or in TEXMFHOME.
> >>
> >> portability issue of format files remains.
> >
> > That decided me: I've dumped my usual files into a format; I'd never
> > really done it because, besides changing my files a lot, I didn't
> > really know how to create a format in TeXLive (a simple shell script
> > is a safeguard against my forgetting again), and I knew Lua code
> > wasn't dumped so I had to tweak all Lua material so that it is both
> > loaded and stored into \everyjob.
> >
> > The gain in time is quite wonderful. I'm even thinking about writing a
> > format for every document, with the relevant fonts preloaded; again a
> > simple script could automate that easily.
>
> That's actually an “old” dream of mine: I'd like to be able to
> automatically produce a format for any given document (up to
> \begin{document}) which is changed if and only if the preamble is
> changed. LuaTeX seems to be the simplest way to do this via some Lua
> action. However, so far I wasn't even able to produce a format dump on
> LuaTeX. (I'm trying to do this by now using the myformat.ltx and
> mylatexformat.ltx, but they produce errors when used with LuaTeX. I
> still have a lot to learn for producing formats, I guess …)

Here's what I've done; since I'm quite lame with shell scripts, and
since everything is executed from my editor (Vim) anyway, it was just
simpler for me to script said editor; it's less useful to anybody
interested, but the code's quite simple so it's easily adapted,
provided one uses a powerful enough editor (the real Vim code is at
the end of this message; here I just describe what the code does --
it has just occurred to me, by the way, that I could have done the
whole thing in Lua, since Vim can be scripted with Lua):

When compiling, the following happens:

1. If <jobname>.fmt already exists in the current directory, compile
with "luatex fmt=<jobname>.fmt <jobname>.tex".
2. If the first non-blank line of <jobname>.tex isn't \format, compile
with the default luatex format or whatever else suits your need.
3. If none of the above is true (i.e. <jobname>.fmt doesn't exist and
the file begins with \format), then:
a) Create a file <jobname>.ini containing whatever basic format you
use, plus the material between \format and \endformat in the
current file, plus a few other things, like "\long\def\format#1\endformat{}",
so that the material isn't processed again when the file is really
compiled.
b) Create the format with "luatex --ini <jobname>.ini".
c) Deletes <jobname>.ini to avoid garbage.
d) Compile with "luatex --fmt=<jobname>.fmt <jobname>.tex".

Point 1 could be changed so that the format is recreated if the
material between \format and \endformat has been modified.

Well, well, that's good fun, but actually I just won half a second (I
don't load TikZ, mind you), and most of the time is spent processing
good ol' TeX macros (which I tend to write pretty thick, admittedly)...

(My saying above that the gain in time was wonderful was due to the
fact that I'd only tested dummy documents with heavy fonts; now with
normal fonts and long documents, it's not so wonderful anymore...)

> >>   > > There is not much text in the PDF files, just a few small tables.
> >>   > > Maybe this is the reason why the font subsetting doesn't take much
> >>   > > time.  The main content of the file are ECG plots created with
> >>   > > \pdfliteral.  BTW, my first attempt was to use Metapost but it was
> >>   > > much too slow.  Processing data with Lua and writing PDF stuff
> >>   > > directly into the PDF file is extremely fast.  It's similar to what
> >>   > > you described in a TUGboat article.  The difference is that your
> >>   > > approach is more Metapost orientated while mine is more PostScript
> >>   > > orientated, just because I don't need a user interface.
> >>   >
> >>   > Nice to know that article rang some bells.
> >>
> >> course, I was interested even more because I was doing something
>
> I was very interested, too, and used the functions provided some times.

Thank you for the remark. Time for me to start a cult :)

Best,
Paul

" Vim script code:

function! MakeFormat ()
" <jobname>.ini
let ini  = expand("%:r") . '.ini'
" <jobname>.fmt
let fmt  = expand("%:r") . '.fmt'

" Format doesn't exist
if !len(glob(fmt))

"  Buffer begins with \format
if getline(nextnonblank(1)) == '\format'

" Base format to be loaded.
let data = ['\let\DUMP\dump \let\dump\relax \input pitex']

" Gathers material between \format and \endformat
let l = nextnonblank(1) + 1
let L = getline(l)
while L != '\endformat'
let l += 1
let L = getline(l)
endwhile
\let\DUMP\undefined \dump')

" Writes <jobname>.ini
call writefile(data, ini)

" Dumps format
echo 'Creating format ' . fmt
call system('luatex --ini ' . ini)

" Deletes <jobname>.ini
call delete(ini)

else " No special format
let fmt = 'luatex'
endif
endif

" Compiles
echo 'Compiling'
call system('luatex --fmt=' . fmt . ' ' . expand('%:r'))
endfunction