[tex-k] Speed of kpsewhich when querying the repertories of the TeX installation

Fri May 1 08:03:35 CEST 2015

Le 1 mai 2015 à 01:05, Norbert Preining <preining at logic.at> a écrit :

>>> If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
>>> you will see what it's doing.
>> 
>> I am sure kpsewhich does a lot, but does one need to do such a lot
>> such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?
> 
> Yes, one has to. Otherwise 20 years of expectation would be broken.
> 
> People regularly redefine TEXMFHOME and TEXMFLOCAL, and that might even happen on multiple levels (distribution makes first change, system admin another).
> 
> So besides:
> * reading the main texmf.cnf
> * searching for other texmf.cnf
> * evaluating their contents
> there is only hard coded path, and that is a no-go.

There must be at least one initial hard-coded path no?
I can't relocate kpsewhich anywhere on my system or can I ?

Are you sure kpsewhich does not do, apart from evaluating the contents of the texmf.cnf
files, do some un-needed parsing of ls-R files, I mean unneeded when kpsewhich is
invoked as kpsewhich --expand-path $TEXMFHOME (or whatever)?

This is why I suggested providing a separate utility to provide only that service of
identifying the locations of the repertories: where is the DIST tree?, where is the LOCAL tree?,
where is the HOME tree? 

It appears that there is no issue if the user has set-up corresponding environment
variables. 

But in that cas TeXLive installation's manual should be explicit about it. I am sure
this brings other complications.

> 
> 
> Let us go back to your Makefile: how many var-values do younormally need? 2? 3? Evaluating them once and saving costs 300ms.
> 
> Evaluating a complex Makefile with rules is probably more time consuming.
> 
> We do this caching of vars in many of our scripts, and that is definitely not the bottle neck.

I did that naturally when I had the problem :

TEXMF_local = $(shell kpsewhich -var-value TEXMFLOCAL)

# as kpsewhich is very slow (.5s) I want to evaluate once only.
installlocal: xint.tds.zip
	$(eval $@_tmp := $(TEXMF_local))
	unzip xint.tds.zip -d $($@_tmp) && texhash $($@_tmp)

> 
> I would invest time in other, that is to say possible, optimizations.

This is not only a Makefile issue

If I resuscitated this issue after some months, it is because I discovered yesterday
that Emacs/AUCTeX 11.88 on Mac OS X  very lengthy launch was entirely
to be explained by its nine calls to kpsewhich.

Earlier AUCTeX 11.86 did only 2, and furthermore it happened during initial
launch of Emacs, not on (first) opening  a .tex file

With 9 calls to kpsewhich, on my (comparatively quite fast, with SSD) MBA with
Mavericks and TeXLive 2014, this meant a 5 seconds wait when first launching AUCTeX 11.88

On an older Mac machine, as the one Adam Maxwell tested my October 2014 findings
about the slowness of kpsewhich --var-value TEXMFHOME, it would have been 15 seconds.

With my machine and TeXLive 2015, we go from 5 seconds to less than 1 second
thanks to the work of Adam Maxwell into improving the Mac OS X speed of
kpsewhich.

Currently, the AUCTeX maintainers are taking into accounts these findings
(this is ongoing) and already yesterday they had reduced from 9 to 5 the number of
kpsewhich calls, thus halving the launch time

I try to get them to make only 1 such call, querying all variables
simultaneously as is allowed by kpsewhich

see

http://lists.gnu.org/archive/html/auctex/2015-04/msg00043.html

and next messages from that thread

Let me explicitely mention to people reading this thread that 
this issue does concern also Linux boxes, but may well be hidden to most
because their TeXLive installation has set-up environment variables
for TEXMFLOCAL, TEXMFHOME, etc...

best

Jean-François

> 
> Norbert
> ------------------------------------------------------------------------
> PREINING, Norbert                              http://www.preining.info
> JAIST, Japan                                 TeX Live & Debian Developer
> GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0  ACF0 6CAC A448 860C DC13
> ------------------------------------------------------------------------
> 
> On May 1, 2015, at 6:51 AM, jfbu <jfbu at free.fr> wrote:
> 
>> 
>> Le 30 avr. 2015 à 23:42, Karl Berry <karl at freefriends.org> a écrit :
>> 
>>>   I may write a Makefile which will act after querying
>>>   the repertories locations. But one tenth of a second
>>>   for each kpsewhich call is a lot.
>>> 
>>> Save the result instead of making the same call thousands of times.  I
>>> don't agree that 100 ms is such a horrible time to do all those
>>> filesystem operations.  In fact, it seems rather quick to me.
>>> 
>>>   Could it be imagined to have some kpsewhichdir utility
>>> 
>>> 1) I fail to see how making a utility restricted to a single variable
>>> could speed anything up.  
>> 
>> 
>> I probably did not express myself precisely. I am not targeting
>> a single variable; I am targeting the variables describing the main
>> repertories of the TeXLive installation
>> 
>> 
>> 
>>> Surely what takes the time is looking for the
>>> config files in all the possible directories, and (more so) reading the
>>> ls-R file.  
>> one might need to do things to find packagefoo.sty via the 
>> parsing of the ls-R files or system calls to ls, but why would these things
>> be needed for the main installation repertories ???
>> 
>> Perhaps many distributions set up environment variables thus
>> the speed penalty is not obvious to these users.
>> 
>> 
>>> 
>>> 2) Regardless, the idea of a utility dedicated to a single variable
>>> sounds wrong in principle to me and I don't think it should be done.
>>> 
>>>   the environment variable $TEXMFHOME 
>>> 
>>> So far as I can see, if the envvar is set, its value is already (has
>>> always been) returned "instantly".  The lookups are only done if they
>>> need to be.
>>> 
>>> karl
>>> 
>> 
>> 
>> I have none such environment variable and do not see why I should
>> set some
>> 
>> Jean-François

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://tug.org/pipermail/tex-k/attachments/20150501/898faeb0/attachment-0001.html>