[luatex] lualatex / includegraphics / open files

[luigi scarso]

On Fri, Dec 1, 2017 at 1:49 PM, Knut Petersen <Knut_Petersen at t-online.de> wrote:
> Hi everybody!
>
> Assume that the files testa.pdf, testb.pdf and testc.pdf exist.
>
> Further assume there is a file mwe.tex:
>
> \NeedsTeXFormat{LaTeX2e}
> \documentclass[12pt]{letter}
> \usepackage{shellesc,graphicx}
> \begin{document}
> \includegraphics{testa.pdf}\newpage
> \includegraphics{testb.pdf}\newpage
> \includegraphics{testc.pdf}\newpage
> \immediate\write18{ lsof | grep latex | grep test | less -S }
> \end{document}
>
> If (on a linux system) you execute
>
> lualatex --shell-escape mwe
>
>
> and
>
> xelatex --shell-escape mwe
>
> you see a difference: the included pdf files are open if you use
> lualatex, they are _not_open_ when xelatex is used.
>
> That difference is not a problem if you want to include only a few
> files, but it is a problem if you want to include a few thousands
> of pdfs - on most sytems there is a limit of about 1024 open files.
>
> That limit can be raised, but at least in my case that does
> not really help: using write18 to start a program when many file
> descriptors are open might expose a bug in the c++ library if "many"
> is "more than 1023". If that child uses execve() to start another child,
> that execve fails with a "*** buffer overflow detected ***" message.
>
> Is this is a problem of luatex? I don't know, but I think it is.
>
> Keeping all files that are included by \includegraphics open is a
> wasteful use of limited system resources (file descriptors) , and xetex
> demonstrates that it is not necessary.
I have tested with 10000 jpg and png, no problem.
In case of a pdf image, it can be that the same reference is used several times
--- e.g  the pdf can have more pages, or you can operate  with
clipping  on the same pdf etc ---
so internally the reference is kept alive until the end of run.
Under linux bash (if you have the rights) you can rise
the maximum number of open file descriptors with ulimit -n
 --- after ulimit -12000 I have tested with  10000 pdf and  it works .
Under Windows 10 you can run a test with 25000 pdf without problem
(bash on windows has the same limit of 1024)

-- 
luigi