[tex-k] Speed of kpsewhich when querying the repertories of the TeX installation

I may write a Makefile which will act after querying
the repertories locations. But one tenth of a second
for each kpsewhich call is a lot.

Save the result instead of making the same call thousands of times. I
don't agree that 100 ms is such a horrible time to do all those
filesystem operations. In fact, it seems rather quick to me.

Could it be imagined to have some kpsewhichdir utility

1) I fail to see how making a utility restricted to a single variable
could speed anything up. Surely what takes the time is looking for the
config files in all the possible directories, and (more so) reading the
ls-R file. If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
you will see what it's doing.

2) Regardless, the idea of a utility dedicated to a single variable
sounds wrong in principle to me and I don't think it should be done.

the environment variable $TEXMFHOME

So far as I can see, if the envvar is set, its value is already (has
always been) returned "instantly". The lookups are only done if they
need to be.

karl

jfbu

2015-04-30 21:51:39 UTC

Post by jfbu
I may write a Makefile which will act after querying
the repertories locations. But one tenth of a second
for each kpsewhich call is a lot.
Save the result instead of making the same call thousands of times. I
don't agree that 100 ms is such a horrible time to do all those
filesystem operations. In fact, it seems rather quick to me.
Could it be imagined to have some kpsewhichdir utility
1) I fail to see how making a utility restricted to a single variable
could speed anything up.

I probably did not express myself precisely. I am not targeting
a single variable; I am targeting the variables describing the main
repertories of the TeXLive installation

Post by jfbu
Surely what takes the time is looking for the
config files in all the possible directories, and (more so) reading the
ls-R file. If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
you will see what it's doing.

I am sure kpsewhich does a lot, but does one need to do such a lot
such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?

one might need to do things to find packagefoo.sty via the
parsing of the ls-R files or system calls to ls, but why would these things
be needed for the main installation repertories ???

Perhaps many distributions set up environment variables thus
the speed penalty is not obvious to these users.

Post by jfbu
2) Regardless, the idea of a utility dedicated to a single variable
sounds wrong in principle to me and I don't think it should be done.
the environment variable $TEXMFHOME
So far as I can see, if the envvar is set, its value is already (has
always been) returned "instantly". The lookups are only done if they
need to be.
karl

I have none such environment variable and do not see why I should
set some

Jean-François

Norbert Preining

2015-04-30 23:05:41 UTC

Post by Karl Berry
If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
you will see what it's doing.

I am sure kpsewhich does a lot, but does one need to do such a lot
such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?

Yes, one has to. Otherwise 20 years of expectation would be broken.

People regularly redefine TEXMFHOME and TEXMFLOCAL, and that might even happen on multiple levels (distribution makes first change, system admin another).

So besides:
* reading the main texmf.cnf
* searching for other texmf.cnf
* evaluating their contents
there is only hard coded path, and that is a no-go.

Let us go back to your Makefile: how many var-values do younormally need? 2? 3? Evaluating them once and saving costs 300ms.

Evaluating a complex Makefile with rules is probably more time consuming.

We do this caching of vars in many of our scripts, and that is definitely not the bottle neck.

I would invest time in other, that is to say possible, optimizations.

Norbert
------------------------------------------------------------------------
PREINING, Norbert http://www.preining.info
JAIST, Japan TeX Live & Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

Post by Karl Berry
I may write a Makefile which will act after querying
the repertories locations. But one tenth of a second
for each kpsewhich call is a lot.
Save the result instead of making the same call thousands of times. I
don't agree that 100 ms is such a horrible time to do all those
filesystem operations. In fact, it seems rather quick to me.
Could it be imagined to have some kpsewhichdir utility
1) I fail to see how making a utility restricted to a single variable
could speed anything up.

I probably did not express myself precisely. I am not targeting
a single variable; I am targeting the variables describing the main
repertories of the TeXLive installation

Post by Karl Berry
Surely what takes the time is looking for the
config files in all the possible directories, and (more so) reading the
ls-R file.

one might need to do things to find packagefoo.sty via the
parsing of the ls-R files or system calls to ls, but why would these things
be needed for the main installation repertories ???
Perhaps many distributions set up environment variables thus
the speed penalty is not obvious to these users.

Post by Karl Berry
2) Regardless, the idea of a utility dedicated to a single variable
sounds wrong in principle to me and I don't think it should be done.
the environment variable $TEXMFHOME
So far as I can see, if the envvar is set, its value is already (has
always been) returned "instantly". The lookups are only done if they
need to be.
karl

I have none such environment variable and do not see why I should
set some
Jean-FranÃ§ois

jfbu

2015-05-01 05:40:40 UTC

Hi

this is interesting thanks, but let me again explicitely state that my only issue
is with querying what TEXMFHOME, TEXMFLOCAL, etc.. are

perhaps this is a no-issue for most people because those are already
set up as environment variables ?

as I will comment more in another mail, the issue is acute with Emacs/AUCTeX 11.88
which (until now where the maintainers on my prompting are starting to issue patches)
queries 9 times kpsewhich during launch. On a fast Mac OS X with TL2014 this meant
a 5 seconds loading time of Emacs /AUCTeX, entirely caused by these 9 calls to
kpsewhich. Current development branch has already divided by 2 this to reduce
to 5 calls caching the earlier results. I am trying to convince the developers to
arrange things to make only 1 call to kpsewhich simultaneously for all variables, as
is possible.

best

Jean-François

There is a way to speed up things.
IIRC most of the time in a kpsewhich call is spent parsing texmf.cnf and reading ls-R files.
Many years ago, I added some code to kpathsea so that internal hash tables be shared among process
to speed up things when tex calls metapost which calls tex etc.
Next Karel Skoupy wrote a kpathsea server exhibiting the same kind of speedup, but surely in a more flexible way.
Maybe some of these ideas could be revived.
Another (simpler) option would be to allow kpsewhich to process several requests, if that can help Jean-François.
Fabrice

Post by Karl Berry
If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
you will see what it's doing.

I am sure kpsewhich does a lot, but does one need to do such a lot
such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?

Yes, one has to. Otherwise 20 years of expectation would be broken.
People regularly redefine TEXMFHOME and TEXMFLOCAL, and that might even happen on multiple levels (distribution makes first change, system admin another).
* reading the main texmf.cnf
* searching for other texmf.cnf
* evaluating their contents
there is only hard coded path, and that is a no-go.
Let us go back to your Makefile: how many var-values do younormally need? 2? 3? Evaluating them once and saving costs 300ms.
Evaluating a complex Makefile with rules is probably more time consuming.
We do this caching of vars in many of our scripts, and that is definitely not the bottle neck.
I would invest time in other, that is to say possible, optimizations.
Norbert
------------------------------------------------------------------------
PREINING, Norbert http://www.preining.info
JAIST, Japan TeX Live & Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

I probably did not express myself precisely. I am not targeting
a single variable; I am targeting the variables describing the main
repertories of the TeXLive installation

Post by Karl Berry
Surely what takes the time is looking for the
config files in all the possible directories, and (more so) reading the
ls-R file.

I have none such environment variable and do not see why I should
set some
Jean-François

--
Fabrice Popineau
-----------------------------
SUPELEC
Département Informatique
3, rue Joliot Curie
91192 Gif/Yvette Cedex
Tel direct : +33 (0) 169851950
Standard : +33 (0) 169851212
------------------------------

jfbu

2015-05-01 06:03:35 UTC

Post by Karl Berry
If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
you will see what it's doing.

I am sure kpsewhich does a lot, but does one need to do such a lot
such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?

There must be at least one initial hard-coded path no?
I can't relocate kpsewhich anywhere on my system or can I ?

Are you sure kpsewhich does not do, apart from evaluating the contents of the texmf.cnf
files, do some un-needed parsing of ls-R files, I mean unneeded when kpsewhich is
invoked as kpsewhich --expand-path $TEXMFHOME (or whatever)?

This is why I suggested providing a separate utility to provide only that service of
identifying the locations of the repertories: where is the DIST tree?, where is the LOCAL tree?,
where is the HOME tree?

It appears that there is no issue if the user has set-up corresponding environment
variables.

But in that cas TeXLive installation's manual should be explicit about it. I am sure
this brings other complications.

Post by Norbert Preining
Let us go back to your Makefile: how many var-values do younormally need? 2? 3? Evaluating them once and saving costs 300ms.
Evaluating a complex Makefile with rules is probably more time consuming.
We do this caching of vars in many of our scripts, and that is definitely not the bottle neck.

I did that naturally when I had the problem :

TEXMF_local = $(shell kpsewhich -var-value TEXMFLOCAL)

# as kpsewhich is very slow (.5s) I want to evaluate once only.
installlocal: xint.tds.zip

Post by Norbert Preining
I would invest time in other, that is to say possible, optimizations.

This is not only a Makefile issue

If I resuscitated this issue after some months, it is because I discovered yesterday
that Emacs/AUCTeX 11.88 on Mac OS X very lengthy launch was entirely
to be explained by its nine calls to kpsewhich.

Earlier AUCTeX 11.86 did only 2, and furthermore it happened during initial
launch of Emacs, not on (first) opening a .tex file

With 9 calls to kpsewhich, on my (comparatively quite fast, with SSD) MBA with
Mavericks and TeXLive 2014, this meant a 5 seconds wait when first launching AUCTeX 11.88

On an older Mac machine, as the one Adam Maxwell tested my October 2014 findings
about the slowness of kpsewhich --var-value TEXMFHOME, it would have been 15 seconds.

With my machine and TeXLive 2015, we go from 5 seconds to less than 1 second
thanks to the work of Adam Maxwell into improving the Mac OS X speed of
kpsewhich.

Currently, the AUCTeX maintainers are taking into accounts these findings
(this is ongoing) and already yesterday they had reduced from 9 to 5 the number of
kpsewhich calls, thus halving the launch time

I try to get them to make only 1 such call, querying all variables
simultaneously as is allowed by kpsewhich

see

http://lists.gnu.org/archive/html/auctex/2015-04/msg00043.html

and next messages from that thread

Let me explicitely mention to people reading this thread that
this issue does concern also Linux boxes, but may well be hidden to most
because their TeXLive installation has set-up environment variables
for TEXMFLOCAL, TEXMFHOME, etc...

best

Jean-François

Post by Norbert Preining
Norbert
------------------------------------------------------------------------
PREINING, Norbert http://www.preining.info
JAIST, Japan TeX Live & Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

I probably did not express myself precisely. I am not targeting
a single variable; I am targeting the variables describing the main
repertories of the TeXLive installation

Post by Karl Berry
Surely what takes the time is looking for the
config files in all the possible directories, and (more so) reading the
ls-R file.

I have none such environment variable and do not see why I should
set some
Jean-François

jfbu

2015-05-01 06:39:13 UTC

[re-formatting my message, sorry]
Hi

this is interesting thanks, but let me again explicitely state
that my only issue is with querying what TEXMFHOME, TEXMFLOCAL,
etc.. are

perhaps this is a no-issue for most people because those are
already set up as environment variables ?

as I will comment more in another mail, the issue is acute with
Emacs/AUCTeX 11.88 which (until now where the maintainers on my
prompting are starting to issue patches) queries 9 times kpsewhich
during launch. On a fast Mac OS X with TL2014 this meant a 5
seconds loading time of Emacs /AUCTeX, entirely caused by these 9
calls to kpsewhich. Current development branch has already divided
by 2 this to reduce to 5 calls caching the earlier results. I am
trying to convince the developers to arrange things to make only 1
call to kpsewhich simultaneously for all variables, as is
possible.

best

Jean-François

Post by Karl Berry
If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
you will see what it's doing.

I am sure kpsewhich does a lot, but does one need to do such a lot
such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?

Yes, one has to. Otherwise 20 years of expectation would be broken.
People regularly redefine TEXMFHOME and TEXMFLOCAL, and that might even happen on multiple levels (distribution makes first change, system admin another).
* reading the main texmf.cnf
* searching for other texmf.cnf
* evaluating their contents
there is only hard coded path, and that is a no-go.
Let us go back to your Makefile: how many var-values do younormally need? 2? 3? Evaluating them once and saving costs 300ms.
Evaluating a complex Makefile with rules is probably more time consuming.
We do this caching of vars in many of our scripts, and that is definitely not the bottle neck.
I would invest time in other, that is to say possible, optimizations.
Norbert
------------------------------------------------------------------------
PREINING, Norbert http://www.preining.info
JAIST, Japan TeX Live & Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

I probably did not express myself precisely. I am not targeting
a single variable; I am targeting the variables describing the main
repertories of the TeXLive installation

Post by Karl Berry
Surely what takes the time is looking for the
config files in all the possible directories, and (more so) reading the
ls-R file.

I have none such environment variable and do not see why I should
set some
Jean-François

jfbu

2015-05-01 06:42:29 UTC

[my reply had too long lines sorry]

Post by Karl Berry
If you run kpsewhich --debug=-1 --expand-path=\$TEXMFHOME,
you will see what it's doing.

I am sure kpsewhich does a lot, but does one need to do such a lot
such to tell where is TEXMFLOCAL and TEXMFHOME and TEXMFDIST ?

There must be at least one initial hard-coded path no? I can't
relocate kpsewhich anywhere on my system or can I ?

Are you sure kpsewhich does not do, apart from evaluating the
contents of the texmf.cnf files, do some un-needed parsing of ls-R
files, I mean unneeded when kpsewhich is invoked as kpsewhich
--expand-path $TEXMFHOME (or whatever)?

This is why I suggested providing a separate utility to provide
only that service of identifying the locations of the repertories:
where is the DIST tree?, where is the LOCAL tree?, where is the
HOME tree?

It appears that there is no issue if the user has set-up
corresponding environment variables.

But in that cas TeXLive installation's manual should be explicit
about it. I am sure this brings other complications.

I did that naturally when I had the problem :

TEXMF_local = $(shell kpsewhich -var-value TEXMFLOCAL)

# as kpsewhich is very slow (.5s) I want to evaluate once only.
installlocal: xint.tds.zip

Post by Norbert Preining
I would invest time in other, that is to say possible, optimizations.

This is not only a Makefile issue

If I resuscitated this issue after some months, it is because I
discovered yesterday that Emacs/AUCTeX 11.88 on Mac OS X very
lengthy launch was entirely to be explained by its nine calls to
kpsewhich.

Earlier AUCTeX 11.86 did only 2, and furthermore it happened
during initial launch of Emacs, not on (first) opening a .tex file

With 9 calls to kpsewhich, on my (comparatively quite fast, with
SSD) MBA with Mavericks and TeXLive 2014, this meant a 5 seconds
wait when first launching AUCTeX 11.88

On an older Mac machine, as the one Adam Maxwell tested my October
2014 findings about the slowness of kpsewhich --var-value
TEXMFHOME, it would have been 15 seconds.

With my machine and TeXLive 2015, we go from 5 seconds to less
than 1 second thanks to the work of Adam Maxwell into improving
the Mac OS X speed of kpsewhich.

Currently, the AUCTeX maintainers are taking into accounts these
findings (this is ongoing) and already yesterday they had reduced
from 9 to 5 the number of kpsewhich calls, thus halving the launch
time

I try to get them to make only 1 such call, querying all variables
simultaneously as is allowed by kpsewhich

see

http://lists.gnu.org/archive/html/auctex/2015-04/msg00043.html

and next messages from that thread

Let me explicitely mention to people reading this thread that this
issue does concern also Linux boxes, but may well be hidden to
most because their TeXLive installation has set-up environment
variables for TEXMFLOCAL, TEXMFHOME, etc...

best

Jean-François

I probably did not express myself precisely. I am not targeting
a single variable; I am targeting the variables describing the main
repertories of the TeXLive installation

Post by Karl Berry
Surely what takes the time is looking for the
config files in all the possible directories, and (more so) reading the
ls-R file.

I have none such environment variable and do not see why I should
set some
Jean-François

Norbert Preining

2015-05-01 07:10:17 UTC

Hi,

Post by Norbert Preining
* reading the main texmf.cnf
* searching for other texmf.cnf
* evaluating their contents
there is only hard coded path, and that is a no-go.

There must be at least one initial hard-coded path no? I can't
relocate kpsewhich anywhere on my system or can I ?

You can relocate, check the installation of TeX Live, you can
choose whereever you want.

kpsewhich searches for the initial texmf.cnf in a (long) list
of defined locations, that allows for relocation, plus some
fixed locations.

Post by jfbu
Are you sure kpsewhich does not do, apart from evaluating the
contents of the texmf.cnf files, do some un-needed parsing of ls-R

Of course it does ls-R parsing, but it is not unneeded. Because
the ls-R is used to find the other files, including other texmf.cnf
files (as far as I know).

So yes of course it parses, but it is not unnecessary.

Post by jfbu
files, I mean unneeded when kpsewhich is invoked as kpsewhich
--expand-path $TEXMFHOME (or whatever)?

And how is kpsewhich supposed to find all texmf.cnf files that might
change the behaviour?

Post by jfbu
This is why I suggested providing a separate utility to provide
where is the DIST tree?, where is the LOCAL tree?, where is the
HOME tree?

The problem is you *CANNOT* decide it without the full parsing.

It is like TeX (well, not really), you need a full parser to
provide the same feature.

Post by jfbu
It appears that there is no issue if the user has set-up
corresponding environment variables.

Of course not, there is always a shortcut in this case, as Karl
stated.

Post by jfbu
But in that cas TeXLive installation's manual should be explicit
about it. I am sure this brings other complications.

For what? How many kpsewhich calls a user does in his *whole*life*,
I am wondering ....

Post by jfbu
TEXMF_local = $(shell kpsewhich -var-value TEXMFLOCAL)
# as kpsewhich is very slow (.5s) I want to evaluate once only.
installlocal: xint.tds.zip

Scrapping .5s from an *install* routine that is called *once*
every whatever, weeks, months?

If you don't have *any* other problem you must be happy.

Try instead improving the TeX parsing speed, that is *much* more
effective support for everyone.

Post by jfbu
If I resuscitated this issue after some months, it is because I
discovered yesterday that Emacs/AUCTeX 11.88 on Mac OS X very
lengthy launch was entirely to be explained by its nine calls to
kpsewhich.

According to your statement, 9 * 0.2 sec is about 1.8sec. Taking
caching of the ls-R from the OS into account, my guess is that
the actual time spent in kpsewhich calls is about 1.5sec.

IF you are so conservative about it, why not add to your .profile:
TEXMFLOCAL=$(kpsewhich -var-value TEXMFLOCAL)
TEXMFHOME=$(kpsewhich -var-value TEXMFHOME)
export TEXFMLOCAL TEXMFHOME
and all will be hell of a speed fast....

Isn't that the simplest solution for all your problems?

Post by jfbu
Earlier AUCTeX 11.86 did only 2, and furthermore it happened
during initial launch of Emacs, not on (first) opening a .tex file
With 9 calls to kpsewhich, on my (comparatively quite fast, with
SSD) MBA with Mavericks and TeXLive 2014, this meant a 5 seconds
wait when first launching AUCTeX 11.88

End after that? 0 .. right. Aehmmm, well.

Post by jfbu
On an older Mac machine, as the one Adam Maxwell tested my October
2014 findings about the slowness of kpsewhich --var-value
TEXMFHOME, it would have been 15 seconds.

15sec??? I would be surprised, that must be a very bad machine that
loading a ls-R file into memory and parsing it takes that much
time.

Post by jfbu
Let me explicitely mention to people reading this thread that this
issue does concern also Linux boxes, but may well be hidden to
most because their TeXLive installation has set-up environment
variables for TEXMFLOCAL, TEXMFHOME, etc...

As the main dev of the TeX Live infra, I am running several installation
of TeX Live, and I do NOT have *any* env variable. As a logician and
mathematician I am editing TeX files in Emcas with AucTeX on a daily
basis, and never had the feeling that *this* is the prbolems.

As said above, there is no fast way unless you do something like
a server. But that needs to monitor the file system for changes
in the texmf.cnf files, and this is something that nobody wants to
do for kpsewhcih calls ...

Norbert

------------------------------------------------------------------------
PREINING, Norbert http://www.preining.info
JAIST, Japan TeX Live & Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

Norbert Preining

2015-05-01 07:18:54 UTC

I have read through the linked thread you provided, and it is *not*
clear to me that the failure is at kpsewhich side. Other people
have responded with no problem.

Could it be that on the Mac somehting else happens, that makes
kpsewhich actually *search* the hard drive instead of using ls-R?
If there are some lonesome // or missing ls-R or whatever,
this can really create problems, but that is not a problem
of kopsewhich.

Norbert

------------------------------------------------------------------------
PREINING, Norbert http://www.preining.info
JAIST, Japan TeX Live & Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
------------------------------------------------------------------------

jfbu

2015-05-01 07:46:42 UTC

I have read through the linked thread you provided, and it is *not*
clear to me that the failure is at kpsewhich side. Other people
have responded with no problem.

Hi Norbert

there were only two other guys apart from me and Tassilo
(and David Kastrup) on this thread.

The OP was on Windows 8 and he has not commented since. I would
be very curious to know what was his issue, which may well be
completely unrelated to the Mac OS X/kpsewhich thing.

The other guy on Mac, I am not sure he really has AucTEX 11.88
He has not provided any usable information.

Again, during the fall of 2014, Adam Maxwell got even worse results
than my 2012 Mac Book Air with Mavericks. And he felt compelled
into profiling kpsewhich and obtained a 6 fold speed improvement.

Post by Norbert Preining
Could it be that on the Mac somehting else happens, that makes
kpsewhich actually *search* the hard drive instead of using ls-R?
If there are some lonesome // or missing ls-R or whatever,
this can really create problems, but that is not a problem
of kopsewhich.

I have vanilla TL2014 or TL2015 with no fiddling of texmf.cnf
files at all, aside from setting up the location of the local
tree and of the home tree and of the home var tree

TEXMFLOCAL = /usr/local/texlive/texmf-local-2014
TEXMFHOME = ~/Library/texmf
TEXMFVAR = ~/Library/texlive/2014/texmf-var
TEXMFCONFIG = ~/Library/texlive/2014/texmf-config

in /usr/local/texlive/2014/texmf.cnf.

Nothing else changed.

Same for the pretest of TL2015.

Recall that Adam Maxwell divided by 6 the running time
of kpsewhich on Mac OS X, thus on TL2015 the situation
is less acute.

But still, 1/10th of a second for each call.

Best

Jean-François

jfbu

2015-05-01 07:38:07 UTC

Post by Norbert Preining
Hi,

Hi Norbert

thanks for the detailed answer

I will only reply to the locations where you refer to
my experience of how long kpsewhich takes to complete

(sorry if I cut-out most of your reply, it was very
informative and now it is too late to paste it back
and I had no comments apart from thanking you for the info)

Post by jfbu
But in that cas TeXLive installation's manual should be explicit
about it. I am sure this brings other complications.

For what? How many kpsewhich calls a user does in his *whole*life*,
I am wondering ....

as mentioned below Emacs/AUCTeX 11.88 does it 9 times per launch

Post by jfbu
TEXMF_local = $(shell kpsewhich -var-value TEXMFLOCAL)
# as kpsewhich is very slow (.5s) I want to evaluate once only.
installlocal: xint.tds.zip

Scrapping .5s from an *install* routine that is called *once*
every whatever, weeks, months?

well, as I said, the issue became really acute to me only since
I realized that Emacs/AUCTeX launch time was affected by it

Post by Norbert Preining
If you don't have *any* other problem you must be happy.

I have the Emacs/AUCTeX problem ! (currently being resolved by
the maintainers)

According to your statement, 9 * 0.2 sec is about 1.8sec. Taking
caching of the ls-R from the OS into account, my guess is that
the actual time spent in kpsewhich calls is about 1.5sec.

from where do you get the 0.2 sec ? It is 0.5 sec on my Mac Book Air
with a 250G SSD and TeXLive 2014

95% of the 5 seconds launching time of AUCTeX is explained by the
nine kpsewhich calls

Post by Norbert Preining
TEXMFLOCAL=$(kpsewhich -var-value TEXMFLOCAL)
TEXMFHOME=$(kpsewhich -var-value TEXMFHOME)
export TEXFMLOCAL TEXMFHOME
and all will be hell of a speed fast....
Isn't that the simplest solution for all your problems?

It is a good solution.

But it would make me deal with issues I am not
to eager to go into, like how does the OS pass
the environment variables to GUI processes.

I never installed Yosemite (Mac OS 10.10) but as far as I know
there was a big bug in it regarding setenv/getenv

https://email.esm.psu.edu/pipermail/macosx-tex/2014-October/053065.html

I also recalled from earlier times that there was a

$HOME/.MacOSX/environment.plist

which at some point disappeared.

http://apple.stackexchange.com/questions/126455/is-home-macosx-environment-plist-still-used-in-os-x-10-9-2-for-setting-global

Thus, regarding environment variables, things fluctuate.

However, it would instantly solved all my problems. But not
the ones for distributing packages with install scripts.

End after that? 0 .. right. Aehmmm, well.

Post by jfbu
On an older Mac machine, as the one Adam Maxwell tested my October
2014 findings about the slowness of kpsewhich --var-value
TEXMFHOME, it would have been 15 seconds.

15sec??? I would be surprised, that must be a very bad machine that
loading a ls-R file into memory and parsing it takes that much
time.

quoting from
https://email.esm.psu.edu/pipermail/macosx-tex/2014-October/053021.html

On a 2007 MacBook Pro with a hard disk, running
TL 2013 on Mac OS X 10.9.5, I get

$ time kpsewhich -var-value=TEXMFLOCAL
/usr/local/texlive/texmf-local

real 0m1.401s
user 0m0.926s
sys 0m0.038s

AUCTeX 11.86 did only 2 calls to kpsewhich

AUCTeX 11.88 does 9 calls.

If 1/10th of a second is not a lot, 9/10th of a second starts
being a nuisance

Post by Norbert Preining
As said above, there is no fast way unless you do something like
a server. But that needs to monitor the file system for changes
in the texmf.cnf files, and this is something that nobody wants to
do for kpsewhcih calls ...

I have the lagging feeling that some other approach, perhaps with
a single file at some mandatory location, could bring some ways
to instantly recover the location of the main trees of a TeX installation

Best

Jean-François

jfbu

2015-05-01 08:01:28 UTC

Post by jfbu
With 9 calls to kpsewhich, on my (comparatively quite fast, with
SSD) MBA with Mavericks and TeXLive 2014, this meant a 5 seconds
wait when first launching AUCTeX 11.88

End after that? 0 .. right. Aehmmm, well.

what do you mean? "first launching AUCTeX" may have been confusing,
it meant "first launch in each Emacs session, i.e. on first C-xC-f
a foo.tex file"

If you have auctex 11.86 installed in a site-start repertory it
well be presumably launched during the Emacs launch itself.
Then there is no delay in opening a .tex file.

However, if you install AUCTeX 11.88 via ELPA, and don't do
anything else to load it, it is only partially loaded via the
package-initialize mechanism, and the bulk of laoding it,
even byte-compiled, is on opening for the first time a .tex file

Thus, for me since november 2014, this meant 5 seconds wait
each time.

And I understood only yesterday this was entirely due to the
kpsewhich calls.

Jean-François

jfbu

2015-05-01 08:07:15 UTC

It is a good solution.

It is not such a good solution: for example currently I am
constantly switching back and forth between TL2014 and TL2015
with a single click, I don't need to source my .bash_profile,
nor do I need to relaunch Emacs, etc...

Jean-François

jfbu

2015-05-01 08:28:04 UTC

Post by jfbu
There must be at least one initial hard-coded path no? I can't
relocate kpsewhich anywhere on my system or can I ?

You can relocate, check the installation of TeX Live, you can
choose whereever you want.

By initial hard-coded path I meant "relative path". Sorry for
my possibly imprecise language.

I was under the impression that kpsewhich location had
to be in $TEXMFDIST/../bin/<hardwarespec>/
but I admit I had not checked that.

Thus I was well aware the whole thing is relocatable, which
made me think there must be a way that kpsewhich had via
hard-coded *relative* paths to fetch at least *one* initial
texmf.cnf

But, I did not think possible to move kpsewhich to /usr/bin/
for example while leaving the other things in the main
TeXLive tree.

Post by Norbert Preining
kpsewhich searches for the initial texmf.cnf in a (long) list
of defined locations, that allows for relocation, plus some
fixed locations.

Sort of what I expected.

Jean-François

jfbu

2015-05-02 15:07:35 UTC

It would add the kspewhich extra-time to the launching time
of all things.

I will complete and finish, in case there are people interested,
my story here.

Emacs/AUCTeX 11.88 called kpsewhich 9 times during launch.

On my 2012 MacBook Air, which whenever I could compare it to anything
else, for example regarding the speed of execution of Python programs,
is reasonably fast and has a 250G SSD, this meant a wait of 5 seconds
with the kpathsea library of TeXLive 2014.

(CPU 2 GHz Intel Core i7, Memory 8 Go 1600 MHz DDR3, 250Go SSD)

Developers of Emacs/AUCTeX have released 11.88.5 which calls
only 3 times kpsewhich during launch.

Thus the loading time has been divided by 3.

Notice that AUCTeX 11.86 did only 2 calls to kpsewhich, not 9,
hence that issue was not easily perceived earlier. Besides,
it is a non-issue for people with environment variables set.

Although kpsewhich from TL2015 will be six times faster
on Mac OS, it is still frustrating to have perhaps still
75% of the launching time of AUCTeX mode explained by these
remaining 3 kpsewhich calls.

Thus I have decided in the end that I will set up environment
variables.

Fortunately I have on my System the TeX Distribution Panel
with its accompanying bunch of symbolic links in /Library/TeX
allowing single-click move from one distribution to another.

The collection and names of symbolic links in the various
/Library/TeX/Distributions/TeXLive-YYYY.texdist
are a bit baroque at times and examining the ones I found
I needed to make a few fixes, but this will solve my problems.

Thus I now have this in my bash_profile:

TEXDISTRIBCONTENTS="/Library/TeX/Distributions/.DefaultTeX/Contents"
export TEXMFSYSVAR="$TEXDISTRIBCONTENTS/TexmfSysVar"
export TEXMFLOCAL="$TEXDISTRIBCONTENTS/TexmfLocal"
# export TEXMFDIST="$TEXDISTRIBCONTENTS/AllTexmf/texmf-dist"
# export TEXMFMAIN="$TEXDISTRIBCONTENTS/AllTexmf/texmf" # -> texmf-dist if >2012
# The above needed for TeXLive2012 and earlier
# but as AUCTeX will not see with TeXLive2013+ that this texmf and texmf-dist
# are in fact the same, the duplicate repertory will not be pruned. I don't
# know how these repertories are used, thus I prefer to drop <2013 support
# Thus:
export TEXMFDIST="$TEXDISTRIBCONTENTS/Root/texmf-dist"
export TEXMFMAIN="$TEXMFDIST"
export SYSTEXMF="$TEXMFSYSVAR:$TEXMFLOCAL:$TEXMFDIST"
export TEXMFHOME="/Users/---/Library/texmf"

This is because SYSTEXMF, TEXMFLOCAL, TEXMDIST, TEXMFMAIN, TEXMFHOME
are the things which are queried by AUCTeX 11.88 via kpsewhich.

With this set-up I don't have to change or re-source my
environment variables, I may switch my current distribution
with a single change of one symbolic link
/Library/TeX/Distributions/.DefaultTeX/Contents
(which is done in GUI via the TeX Dist Pane)
and the environment variables need no adjustments.

Only /usr/texbin is added to the PATH, and /usr/texbin is itself
a symbolic link whose ultimate destination depends on the current
/Library/TeX/Distributions/.DefaultTeX/Contents
symlink.

Jean-François

Akira Kakuto

2015-05-02 16:13:25 UTC

Hi,

Post by jfbu
Although kpsewhich from TL2015 will be six times faster
on Mac OS, it is still frustrating to have perhaps still
75% of the launching time of AUCTeX mode explained by these
remaining 3 kpsewhich calls.

If you make your own command kpsevarinfo by replacing
cnf.c by cnf2.c, where in cnf2.c kpathsea_init_db (kpse);
in the line 250 is omitted,
kpsevarinfo -var-value=TEXMFHOME
may be about 10 times faster than
kpsewhich -var-value=TEXMFHOME

Note that kpsevarinfo may be usable only to obtain
information on variables.

Best,
Akira

jfbu

2015-05-02 17:30:05 UTC

Post by Akira Kakuto
Hi,

Post by jfbu
Although kpsewhich from TL2015 will be six times faster
on Mac OS, it is still frustrating to have perhaps still 75% of the launching time of AUCTeX mode explained by these remaining 3 kpsewhich calls.

If you make your own command kpsevarinfo by replacing
cnf.c by cnf2.c, where in cnf2.c kpathsea_init_db (kpse);
in the line 250 is omitted,
kpsevarinfo -var-value=TEXMFHOME
may be about 10 times faster than
kpsewhich -var-value=TEXMFHOME
Note that kpsevarinfo may be usable only to obtain
information on variables.
Best,
Akira

Thanks, looks promising indeed. Might be exactly what I was asking.

I am currently compiling. I looked at README2.building
but couldn't identify in the configure the option which would
allow me to compile kpsewhich separately, I launched ./Build

I will report later on timing

Best,
Jean-François

jfbu

2015-05-02 17:57:23 UTC

Post by Akira Kakuto
Hi,

Post by jfbu
Although kpsewhich from TL2015 will be six times faster
on Mac OS, it is still frustrating to have perhaps still 75% of the launching time of AUCTeX mode explained by these remaining 3 kpsewhich calls.

Thanks, looks promising indeed. Might be exactly what I was asking.
[...]
I will report later on timing

Dear Kakuto-san

ttys005:~ (19:50)$ time kpsevarinfo --var-value=TEXMFHOME
/Users/---/Library/texmf

real 0m0.006s
user 0m0.002s
sys 0m0.002s
ttys005:~ (19:50)$ time kpsevarinfo --var-value=TEXMFLOCAL
/usr/local/texlive/texmf-local-2015

real 0m0.006s
user 0m0.002s
sys 0m0.002s

to be compared with

ttys005:~ (19:50)$ time kpsewhich --var-value=TEXMFHOME
/Users/---/Library/texmf

real 0m0.086s
user 0m0.077s
sys 0m0.007s
ttys005:~ (19:51)$ time kpsewhich --var-value=TEXMFLOCAL
/usr/local/texlive/texmf-local-2015

real 0m0.087s
user 0m0.078s
sys 0m0.007s

very promising !

what are the restrictions if any ? by

Post by Akira Kakuto
Note that kpsevarinfo may be usable only to obtain
information on variables.

do you mean simply that one can no use it for anything else
but querying values of the variables, but that these is
done with full functionality ?

best

Jean-François

jfbu

2015-05-02 18:20:09 UTC

Post by jfbu
TEXDISTRIBCONTENTS="/Library/TeX/Distributions/.DefaultTeX/Contents"
export TEXMFSYSVAR="$TEXDISTRIBCONTENTS/TexmfSysVar"
export TEXMFLOCAL="$TEXDISTRIBCONTENTS/TexmfLocal"
[...]

But with such environment variables the log files of
tex/latex/etc.. compilations now display the faked
symlinked paths, not the absolute physical ones.

I am now with (perhaps not efficient, I am no proficient
bash user)

TEXDISTRIBCONTENTS="/Library/TeX/Distributions/.DefaultTeX/Contents"
export TEXMFSYSVAR="`cd -P $TEXDISTRIBCONTENTS/TexmfSysVar && pwd`"
export TEXMFLOCAL="`cd -P $TEXDISTRIBCONTENTS/TexmfLocal && pwd`"
export TEXMFDIST="`cd -P $TEXDISTRIBCONTENTS/Root/texmf-dist && pwd`"
export TEXMFMAIN="$TEXMFDIST"
export SYSTEXMF="$TEXMFSYSVAR:$TEXMFLOCAL:$TEXMFDIST"
export TEXMFHOME="/Users/---/Library/texmf"

I must restart a shell after switching distributions.

Jean-François

Reinhard Kotucha

2015-05-04 07:41:10 UTC