[luatex] detailed 'diagnostic' of MikTeX/Lua*/fontspec issues

[Evan Cooch]

Cross-posting to both the LuaTeX discussion (since the problems I and a 
few others have had *may* be lua* related), and the MikTeX list (or they 
may be unique to MikTeX installs), or perhaps an interesting/unfortunate 
interaction of the two.  Apologies for the breach of cross-post 
etiquette, but I have my (stated) reasons. Long post, but wanted to 
document everything -- basic motivation is that recent updates to MikTeX 
install cause no shortage of problems for compiles with lua*, at least 
for some folks (regular LaTeX compilation fine).  If you want to save 
time, jump down to 'step 12'. This is where the problem lies -- and it 
is directly related to 'fontspec'.

Step 1\ take one of my laptops - Windows 7 Pro, fully patched.  Has 
Adobe Pro 10 instaqlled (the full package, not simply the reader). Do a 
complete uninstall of MikTeX 2.9. Clean out *everything* -- all traces, 
from c:\program files (x86), registry, c:\users appdata - the whole 
works.  Run CCleaner to even clobber any temp files that might be 
lingering. Heck, even defragged the drive while I was at it. ;-)

Step 2\ Reboot machine. Download fresh copy of MikTeX installer 
(downloaded 6-29-2013) - although the machine is running 64-bit Windows, 
I use the 32-bit installer (never had a reason not to). Now, on this 
machine, I am a user and admin -- meaning, I have full administrative 
rights.

Step 3\ start the installer. Accept the MikTeX copying conditions. Now, 
the step which seems to generate a far bit of 'debate' amongst MikTeX 
users - install for anyone who uses the computer, or only for egc. In 
all the other times I've installed MikTeX, over *many* years, I've never 
done other than selecty install for 'anyone' - even though on ~90% of 
the machines I am the only user.  Install to default C:\Program Files 
(x86)\MikTeX 2.9.  Install proceeds entirely without incident.

Step 4\   Now for a MWE, use the following:

      \documentclass[11pt,letterpaper]{article}

      \usepackage{fontspec}
         \setmainfont{XITS}
      \usepackage{unicode-math}
         \setmathfont{XITS Math}

      \begin{document}

      Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent 
in tincidunt nulla. Donec quis laoreet est. Aenean vitae est felis, a 
venenatis magna. Mauris semper bibendum vulputate. Etiam bibendum quam 
ut erat rutrum consequat.

      \begin{equation}
          \frac{dN}{dt}=rN\left(1-\frac{N}{K}\right)
      \end{equation}

      \end{document}


To test compile with *regular* LaTeX, (or, to text LuaLaTeX without 
fontspec) I simply comment out the following:

\usepackage{fontspec}
\setmainfont{XITS}
\usepackage{unicode-math}
\setmathfont{XITS Math}


Step 5\ After the basic MikTeX install, I install the following packages 
(needed by the MWE, above):  XITS fonts, unicode-math

Otherwise, nothing more than the basic MikTeX install (i.e., for now, I 
don't add any other packages).  Now, one nuance is that XITS gets 
installed in

C:\Program Files (x86)\MiKTeX 2.9\fonts\opentype\public\xits


For a variety of reasons, luaotfload doesn't 'find' fonts in TEXMF, so 
you need to hack lualibs-dir.lua, as per

http://tex.stackexchange.com/questions/47156/luaotfload-cant-find-fonts

Basically, you add

P("(")  / "%%(" +
P(")")  / "%%)" +

to the file, around line 95. The brackets will be escaped and therefore 
the fonts in the "program files (x86)/miktex" folder can be found.I've 
had to do this on *all* my other machines with MikTeX (6 different 
machines). Has worked perfectly on all of them.


Step 6\ now, ready for first compile of MWE.  I have found in the past 
that for a first compile of anything with LuaLaTeX, I need to do the 
compile from the command line. My normal editor is WinEdt, and for some 
reason, some of the things (other packages) the LuaLaTeX needs don't 
install 'on the fly' from within WinEdt. This isn't an issue. The MWE is 
trivial, and doing everything from the CLI eliminates weird editor 
things from the larger mix of things. So, CLI it is.

Start with simple LaTeX compile of the MW, after commenting out the 
'fontspec stuff' (see step 4, above). Works fine. DVI is created. Looks 
just like expected - cmr fonts, equation rendere3d properly etc.

Now, do a LuaLaTex compile on the MWE, again after commenting out the 
'fontspec stuff'.  Again, no problems. The PDF is created, and looks 
exactly like it should.

Now, for the big test -- uncomment the 'fontspec stuff' -- in other 
words, try to pull in XITS fonts for text and maths.  Run

lualatex mwe.tex from CLI

MikTeX asks to install expl3.sty   -- done.

Next, MikteX asks for xparse.sty  -- done

Finally, MikTeX asks for t3enc.def  -- done

(Note for MikTeX users: it only asks for these installs once, obviously, 
but if you try this through WinEdt, it fails -- hence CLI compile first 
time).

Now --- luaotfload starts grinding away -- first says Font names 
database not found (of course, since not run before), so it proceeds to 
create a new one -- scanning TEXMF and OS fonts.

Grinds along a bit, then MikTeX asks to install  filehook.sty -- done.  
Then it asks for lualatex-math.sty -- done.  Next, etoolbox.sty -- 
done.  And finally, supp-pdf.sty -- done.

Finally, complete -- PDF created, and looks perfect -- XITS fonts, text 
and equation perfect.



So -- CONCLUSION (1) -- at this point, the current MikTeX installer, and 
files contained within, work perfectly.

Now, for the 'problematic' updates.

step 7\  fire up the update utility for MikTeX (I'll use the GUI-based 
version, rather than mpm from CLI).  Because I installed for 'everyone', 
I have 2 choices: update/Admin, or update/user. I use the Admin flavour. 
After a few moments, it responds telling me that there are *lots* of 
updates, but some are 'hard-wired' (i.e., must be done before you can do 
any other updates). These include

MikTeX-bin-2.9
miktex-cairo-2.9
miktex-etex-base
miktex-graphite2-bin-2.9
xetexurl

(and, importantly for this 'story')

miktex-lua51-bin-2.9 (which is marked for removal)
miktex-lua52-bin-2.9 (which will be installed in its place)

I go ahead and accept these updates.

step 8\ now, from what I've read, there have been issues/concerns for 
some folks for having to rebuild lua* .fmt files. I can either do that 
using the GUI settings tool - if I select the 'formats' tab, and 
re-build lualatex and luatex formats (which are 'excluded' by default 
from the one-click rebuild formats), or I can jump to the CLI and run 
initexmf --dump=lualatex (or initexmf --dump=luatex). Since there should 
be no harm at any point in rebuilding .fmt files, I'll do it now, using 
the CLI.  No error messages reported.

step 9\ now, try the same MWE compile sequence -- first, lualatex 
compile of MWE after commenting out the 'fontspec stuff'.  Works 
perfectly (yes!)

Now...the big fish. lualatex compile of the MWE *with* the 'fontspec 
stuff'.  MikTeX starts off, then asks to install etex.sty -- ty to, but 
it complains that etex.sty is not found. I've seen this before, so I 
jump out to the MikteX GUI-based package manager.  For MikTeX users, you 
need to look for MikTeX-etex-base. There are two packages showing -- one 
labelled miktex-etex=base-2.9 (which is a newer build), so I select that 
one to install.  Then, back out to cmd window to re-try lualatex compile 
of MWE

lualatex mwe.tex

and....perfect! (yes!) And, it works if I try the compile from within 
WinEdt, too (yes x 10!)

So -- CONCLUSION (2) -- at this point, the current MikTeX installer, and 
files contained within, work perfectly, even after the updates to basic 
lmiktex-lua52-bin-2.9  files


step 10\ now, consider the rest of the 'updates'. This might be where 
things get interesting. Loking at the list of updates, there are a bunch 
which are related to 'the compile proces using lua*':

fontspec
lualibs
luamplib
luaotfload
luatexbase

Since the updates all are likely to interact with each other, I'll 
simply install all of them. Now, I anticipate that I'll need to 
re-implement the escape tweak noted in step 5 (above), but that is 
trivial. So, install the updates, and then because in inherent anal 
compulsiveness, reboot the machine.

step 11\ the big moment. Start with lualatex compile of MWE *without* 
the 'fontspec stuff'.

Uh-oh.   (Fatal format file error; I'm stymied).

OK -- so I need to recreate the lualatex .fmt file (I think...).  I 
re-run initexmf --dump=lualatex.  Takes a lot longer to run this time, 
but no reported errors.

Try the lualatex compile again -- success! PDF created. Looks exactly as 
it should (cmr fonts, text and equation fine).

step 12\ Now, before I try the compile of the MWE *with* the 'fontspc 
stuff', I jump out to have a look at lualibs-dir.lua, which I suspect 
has been over-written by a 'new' version.  Suspicion confirmed -- and, 
the lines that need to be tweaked/patched are now further down in the 
file (around line 153).  So, make the tweak...and try the lualatex 
compile of MWE with 'fontspec stuff' included.

And, here is where it all fails -- get the following error message:

luaotfload | db: Font names database no found, generating a new one.
   Ths can take sseveral minutes; please be patient.
lualatex: Unexpected condition.

So, to pursue further, I try running
luaotfload-tool --update --force --verbose=5


But, this fails too. Terminates with 'unexpected condition'. But, I 
think this is spurious since it bombs when it gets to my working 
directory. It seems to grind through TEXMF and OS just fine.


But, if I try luaotfload-tool -vv, it doesn't actually print out any 
loaded fonts.  luaotfload-tool -f doesn't do anything either.


Thinking my 'tweak' to lualibs-dir.lua might be the culprit, I revert to 
the original file.

Alas, no happiness.  I even tried some fonts that are in the OS font 
subdirectory -- makes no difference.

After 3-4 other 'experiments', I conclude...


So -- final CONCLUSION (3) -- everything works, updates and all, so long 
as you don't use fontspec.  Further, luaotfload-tool is either not 
working, or isn't playing nice with something.  Regardless, something 
broken, somewhere.

Which is a shame, since quite frankly, the only reason I and a lot of 
folks use lualatex at all is...fontspec. ;-)


Again, thanks for indulging a *very* long post. If there are particular 
folks who should see this who aren't on either the luatex.org or MikteX 
lists, please forward their way.

Thanks again, and I'm happy to be told of a single CLI command that will 
fix everything, even if it make me looks foolish (as I get older, I'm 
increasingly comfortable with public acknowledgement of what things I 
don't know...).