[tex-k] speed of kpsewhich to get value of TEXMFLOCAL

[jfbu]

Hi,

in a Makefile I am currently testing I have something like this

TEXMF_LOCAL = $(shell kpsewhich -var-value TEXMFLOCAL)
TEXMF_HOME = $(shell kpsewhich -var-value TEXMFHOME)

however evaluation of each variable is very slow, of the order
of half a second on my mac os x mavericks with TL2014

======Makefile:
# TEXMF_LOCAL = `kpsewhich -var-value TEXMFLOCAL`
TEXMF_LOCAL = $(shell kpsewhich -var-value TEXMFLOCAL)
TEXMF_HOME = $(shell kpsewhich -var-value TEXMFHOME)

all: testlocal testhome

testlocal:
@echo $(TEXMF_LOCAL)

testhome:
@echo $(TEXMF_HOME)

======running it
$ time make
/usr/local/texlive/texmf-local
/Users/---/Library/texmf

real 0m0.996s
user 0m0.964s
sys 0m0.026s

What can I do to get TEXMFLOCAL ?
It is not `shell` the culprit as for example
one can test with $(shell ls -l Makefile) to
get execution time without kpsewhich.

I use it only for certain targets, and putting
TEXMF_LOCAL := ...
at the top of the Makefile, as I was going to do,
is currently excluded as it slows down all targets

Jean-Francois

[jfbu]

Post by jfbu
What can I do to get TEXMFLOCAL ?
It is not `shell` the culprit as for example
one can test with $(shell ls -l Makefile) to
get execution time without kpsewhich.

and more directly, simply

$ time kpsewhich -var-value TEXMFLOCAL
/usr/local/texlive/texmf-local

real 0m0.490s
user 0m0.476s
sys 0m0.011s

I realize that some other operations of kpsewhich
have that overhead:
$ time kpsewhich hyperref.sty
/usr/local/texlive/2014/texmf-dist/tex/latex/hyperref/hyperref.sty

real 0m0.499s
user 0m0.484s
sys 0m0.011s

This is on mac os x mavericks, up-to-date TL2014

Jean-Francois

[Andrew Komornicki]

Hi,

would it not be simpler to just define these variables in your
environment so that your makefile can pick them up at will. You would
not have to invoke kpsewhich and incur the execution penalty. Karl most
probably has the most insight into execution bottlenecks with his
routine. just a thought.

regards,
andrew

Andrew Komornicki
Department of Chemistry
Stanford University

Post by jfbu
Hi,
in a Makefile I am currently testing I have something like this
TEXMF_LOCAL = $(shell kpsewhich -var-value TEXMFLOCAL)
TEXMF_HOME = $(shell kpsewhich -var-value TEXMFHOME)
however evaluation of each variable is very slow, of the order
of half a second on my mac os x mavericks with TL2014
# TEXMF_LOCAL = `kpsewhich -var-value TEXMFLOCAL`
TEXMF_LOCAL = $(shell kpsewhich -var-value TEXMFLOCAL)
TEXMF_HOME = $(shell kpsewhich -var-value TEXMFHOME)
all: testlocal testhome
@echo $(TEXMF_LOCAL)
@echo $(TEXMF_HOME)
======running it
$ time make
/usr/local/texlive/texmf-local
/Users/---/Library/texmf
real 0m0.996s
user 0m0.964s
sys 0m0.026s
What can I do to get TEXMFLOCAL ?
It is not `shell` the culprit as for example
one can test with $(shell ls -l Makefile) to
get execution time without kpsewhich.
I use it only for certain targets, and putting
TEXMF_LOCAL := ...
at the top of the Makefile, as I was going to do,
is currently excluded as it slows down all targets
Jean-Francois

[jfbu]

Hello Andrew

Post by Andrew Komornicki
would it not be simpler to just define these variables in your
environment so that your makefile can pick them up at will.

Sorry for not providing background info:

I am preparing a Makefile for a future update
to a package of mine on CTAN. This Makefile will have
targets "installhome" and "installlocal" for the user
with a TeX installation offering TEXMFHOME and TEXMFLOCAL.

Thus, it is not for personal use, or rather those two
targets are not the ones useful to me the author.

I wrote the
Makefile to automatize production of my package.tds.zip
and package.zip for CTAN submission. (currently 10 files
*.sty, 3 pdf's, 2 html's, 1 plain text file, the .dtx,
the Makefile, and the .ins file which is not needed anyhow
as etex package.dtx extracts all).

best,

Jean-Francois

[Karl Berry]

1) Defining environment variables often causes all kinds of trouble down
the road -- stale envvar settings is the #1 culprit in TeX bug reports.
So I don't recommend it.

2) The "bottleneck" is not generic to any installation. Running
kpsewhich -var-value=TEXMFLOCAL returns instantly on my machine, and, I
would assume, just about everyone else's or this exact report would be
flooding the list. After all, this invocation is just reading a couple
relatively small text files.

3) I have no knowledge as to what can be causing the absurd delay on
Mavericks. I don't use Macs. Perhaps something (perhaps environment
variables) is causing kpsewhich to actually search the disk for
everything instead of just reading the ls-R files. Running
kpsewhich --debug=1 --var-value=TEXMFLOCAL >&/tmp/out
and then perusing the (voluminous) output might yield some clues.
I don't know. It's not going to come for free.

4) You could try asking mactex-support at tug.org. Assuming you are using
MacTeX and not macports or any of the other various TeX distributions on
Macs (sorry if you already stated this, I can't remember and can't look
it up right now). Each Mac thing should have its own support forums.

Good luck,
Karl

[Reinhard Kotucha]

Post by Karl Berry
2) The "bottleneck" is not generic to any installation. Running
kpsewhich -var-value=TEXMFLOCAL returns instantly on my machine,

What is "instantly" in seconds? See below.

Post by Karl Berry
and, I would assume, just about everyone else's or this exact
report would be flooding the list. After all, this invocation is
just reading a couple relatively small text files.

It's reading all the ls-R files. Checked with strace(1).
I suppose that kpathsea creates the hash tables too which takes
additional time.

But why are the ls-R files needed in order to determine the value of
TEXMFLOCAL?

| $ time kpsewhich -var-value=SELFAUTOLOC
| /usr/local/texlive/2014/bin/x86_64-linux
|
| real 0m0.003s
| user 0m0.000s
| sys 0m0.002s
|
| $ time kpsewhich -var-value=TEXMFLOCAL
| /usr/local/texlive/texmf-local
|
| real 0m0.189s
| user 0m0.175s
| sys 0m0.011s

Why are SELFAUTOLOC and TEXMFLOCAL treated differently?


BTW, without the filesystem cache I get

| # echo 1 > /proc/sys/vm/drop_caches
|
| $ time kpsewhich -var-value=TEXMFLOCAL
| /usr/local/texlive/texmf-local
|
| real 0m0.555s
| user 0m0.160s
| sys 0m0.021s

Jean-Francois, this value is close to that one you reported. It's quite
useful to run a system monitor when doing benchmarks. The first thing
you'll learn is that benchmarks are more or less worthless when a
resource greedy web browser is running.

Post by Karl Berry
$ time kpsewhich hyperref.sty

In this case there is no overhead. If you use kpsewhich in order to
locate files then all the ls-R files have to be read. This takes more
time than expanding a variable, of course. The question is why the
ls-R files are read in order to determine the value of TEXMLOCAL while
it's not necessary in order to expand SELFAUTOLOC.

Regards,
Reinhard

[Akira Kakuto]

Dear Reinhard,

Post by Reinhard Kotucha
Why are SELFAUTOLOC and TEXMFLOCAL treated differently?

SELFAUTOLOC, SELFAUTODIR, SELFAUTOPARENT, and SELFAUTOGRANDPARENT
are putenved at runtime, while it requires to read texmf.cnf's to
obtain TEXMFLOCAL. SELFAUTOGRANDPARENT is new in TeX Live 2014.

Thanks,
Akira

[jfbu]

Post by Reinhard Kotucha
But why are the ls-R files needed in order to determine the value of
TEXMFLOCAL?
| $ time kpsewhich -var-value=SELFAUTOLOC
| /usr/local/texlive/2014/bin/x86_64-linux
|
| real 0m0.003s
| user 0m0.000s
| sys 0m0.002s
|

I get
$ time kpsewhich -var-value=SELFAUTOLOC
/usr/local/texlive/2014/bin/x86_64-darwin

real 0m0.005s
user 0m0.001s
sys 0m0.002s

although on first invocation it was
real 0m0.019s
user 0m0.001s
sys 0m0.003s

(the 0.005s is on the third invocation)

time kpsewhich -var-value=TEXMFLOCAL
always returns about 0.5s, independently of
the number of invocations

Post by Reinhard Kotucha
BTW, without the filesystem cache I get
| # echo 1 > /proc/sys/vm/drop_caches
|
| $ time kpsewhich -var-value=TEXMFLOCAL
| /usr/local/texlive/texmf-local
|
| real 0m0.555s
| user 0m0.160s
| sys 0m0.021s
Jean-Francois, this value is close to that one you reported. It's quite
useful to run a system monitor when doing benchmarks. The first thing
you'll learn is that benchmarks are more or less worthless when a
resource greedy web browser is running.

I have loads to learn before becoming a power user... I am sure mac os x
provides all means to analyze things (it has an Activity Monitor which
provides all sorts of diagnostics, and surely can be triggered from command
line too) inclusive of the standard Unix tools on which it is built. I am just
ignorant, although I do have also XCode installed on my machine.

For info, I ran again
time kpsewhich -var-value=TEXMFLOCAL
with all
GUI applications off and the result is the same.

Post by Reinhard Kotucha

Post by jfbu
$ time kpsewhich hyperref.sty

In this case there is no overhead. If you use kpsewhich in order to
locate files then all the ls-R files have to be read.

I see, the timing being only a tiny bit smaller on my machine
for -var-value TEXMFLOCAL
compared to locating hyperref.sty
it does suggest that most of what is done in this execution of kpsewhich
for locating a file is also done when running kpsewhich with
only the option -var-value

Post by Reinhard Kotucha
This takes more
time than expanding a variable, of course. The question is why the
ls-R files are read in order to determine the value of TEXMLOCAL while
it's not necessary in order to expand SELFAUTOLOC.
Regards,
Reinhard
--

thanks,

Jean-Francois

[jfbu]

Hi Karl,

Post by Karl Berry
1) Defining environment variables often causes all kinds of trouble

The place where I need TEXMFLOCAL and TEXMFHOME is a Makefile
prepared for the users of my package (see my mail to
Andrew Komornicki), thus setting environment variables was
not an option anyhow.

Post by Karl Berry
2) The "bottleneck" is not generic to any installation.

ah! this is good to know, because I only have access to one other
machine, which is under Linux/Fedora, and did not yet
check the issue there yet. I will do it soon.

Post by Karl Berry
3) I have no knowledge as to what can be causing the absurd delay on
Mavericks. I don't use Macs. Perhaps something (perhaps environment
variables) is causing kpsewhich to actually search the disk for
everything instead of just reading the ls-R files. Running
kpsewhich --debug=1 --var-value=TEXMFLOCAL >&/tmp/out
and then perusing the (voluminous) output might yield some clues.
I don't know. It's not going to come for free.

Thanks, I will try this, but can not report until a day or two because of
tight schedule now

Post by Karl Berry
4) You could try asking mactex-support at tug.org. Assuming you are using
MacTeX and not macports or any of the other various TeX distributions on
Macs (sorry if you already stated this, I can't remember and can't look
it up right now). Each Mac thing should have its own support forums.

I am using MacTeX, thus indeed I could try asking Richard Koch http://www.tug.org/twg/mactex/

Also maybe I could ask Adam Maxwell, the creator of TeX Live Utility
(GUI interface to tlmgr on Mac OS X.)

best,

Jean-Francois

[jfbu]

Post by jfbu
ah! this is good to know, because I only have access to one other
machine, which is under Linux/Fedora, and did not yet
check the issue there yet. I will do it soon.

for what it's worth, on a Linux box with TL2010:

$ time kpsewhich -var-value TEXMFLOCAL
/---/texlive2010/texlive/2010/../texmf-local

real 0m0.069s
user 0m0.057s
sys 0m0.003s

$ time kpsewhich hyperref.sty
/home/---/texmf/tex/latex/hyperref/hyperref.sty

real 0m0.087s
user 0m0.058s
sys 0m0.005s

$ time kpsewhich graphics.sty
/---/texlive2010/texlive/2010/texmf-dist/tex/latex/graphics/graphics.sty

real 0m0.103s
user 0m0.055s
sys 0m0.010s

(the "real" times fluctuate but "user" are more stable)

This is with
$ kpsewhich --version
kpathsea version 6.0.0
Copyright 2010 Karl Berry & Olaf Weber.

TexLive 2010 on a Fedora box,
server is a dual processor Intel Core Duo CPU E8400 @3.00GHz
with 3.4Gi of memory

Jean-Francois