[luatex] lualatex / includegraphics / open files

[Hans Hagen]

On 12/3/2017 10:19 AM, Knut Petersen wrote:
> Am 01.12.2017 um 15:44 schrieb Ulrike Fischer:
>> This has also already been discussed in tex.sx
>>
>> https://tex.stackexchange.com/questions/258127/why-lualatex-gives-erorr-too-many-open-files-but-pdflatex-does-not-on-same-lat
> 
> Thanks for that link.
> 
> I tested pdflatex with a 100.000 lines input and found that pdftex 
> correctly included all the 14.000+ pdfs it was expected to include.
> 
> So
> 
>   * pdflatex works correctly,
>   * xelatex breaks after 5.000 included pdfs,
>   * lualatex breaks after whatever the soft limit for file descriptors
>     is set to.
> 
> luatex is the only engine that keeps all included pdfs open, and this is 
> known since at least 2015.
> I think this is a serious regression compared to pdflatex.
When an image is included a reference node is put on the page and as the 
dimensions have to be known the image is analyzed. The (pdf) file is 
then kept open.

When a page is shipped out the part of the opened file that makes sense 
in included. This can be any (number of) page(s).

When the image is inluded at the tex end the file is kept open. In 
pdftex \immediate is (or at least was) the drive to force a close.

When the image is included at the lua end then file is closed unless 
flagges as to be kept open.

As luatex can be used to include multiple pages it makes sense to keep 
the file open. Inlcuding 1000 pages from a file takes 35 seconds when we 
close the file, and 1 second when we keep it open. So, we keep it open.

We've been experimenting a bit and now consider the following. We always 
close the file after a page has been included. Of course when one puts 
100K images on one page one still can hit the limits of the operating 
system but such it is. (I'm not sure what pdftex does when one puts 100K 
images on one page.)

In addition there will be an option to keep the file open at the tex 
end. So in the end we will have

\immediate\useimageresource{foo.pdf}%
           \saveimageresource         \lastsavedimageresourceindex % closed
           \useimageresource{foo.pdf}%
           \saveimageresource         \lastsavedimageresourceindex % 
kept open
           \useimageresource{foo.pdf}%
           \saveimageresource keepopen\lastsavedimageresourceindex % 
kept open

\directlua{img.write(img.scan{ file = "foo.pdf" })}                  % 
closed
\directlua{img.write(img.scan{ file = "foo.pdf", keepopen = true })} % 
kept open

Of course when you want speed in including more than a handful of pages 
from the same file you're off worse. But that's probbably better than 
this topic popping up every few years. Of course using a mixed approach 
for the same file is not our concern.

Hans



-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------