<div dir="ltr"><div><br></div><div>I have zero expertise, but a faster local [but imperfect] latex compile would be of value to me, too. if TUG were to put a bounty on this feature, I would donate to it, too. Maybe so would others.</div><div><br></div><div>some observations: [1] if state information was saved on each page start, not only would a local page compile be possible, but so would multi-core processing. [2] if more information is stored in the file, maybe storing parts of the latex source would also make a later parsing/decompile of a pdf file, as in pandoc, easier. [3] I think pdftex is no longer the state of the art. if this were to be done, it should be on the luatex engine, not the pdftex engine. [4] I believe this is *not* a trivial task at all. there is a large risk that it will not be completed.<br></div><div><br></div><div> regards, /iaw</div><div><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><span style="font-size:small">--</span></div><div dir="ltr"><span style="font-size:small">Ivo Welch (</span><a href="mailto:ivo.welch@ucla.edu" style="color:rgb(17,85,204);font-size:small" target="_blank">ivo.welch@ucla.edu</a><span style="font-size:small">)</span><br style="font-size:small"><br></div></div></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 15, 2022 at 3:01 AM <<a href="mailto:pdftex-request@tug.org">pdftex-request@tug.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send pdftex mailing list submissions to<br>
<a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://tug.org/mailman/listinfo/pdftex" rel="noreferrer" target="_blank">https://tug.org/mailman/listinfo/pdftex</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:pdftex-request@tug.org" target="_blank">pdftex-request@tug.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:pdftex-owner@tug.org" target="_blank">pdftex-owner@tug.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of pdftex digest..."<br>
Today's Topics:<br>
<br>
1. Re: Caching intermediate compilation results for<br>
near-real-time PDF re-renders during editing (Jamie Vicary)<br>
2. Re: Caching intermediate compilation results for<br>
near-real-time PDF re-renders during editing (Doug McKenna)<br>
3. Re: Caching intermediate compilation results for<br>
near-real-time PDF re-renders during editing (Jamie Vicary)<br>
<br><br><br>---------- Forwarded message ----------<br>From: Jamie Vicary <<a href="mailto:jamie.vicary@cl.cam.ac.uk" target="_blank">jamie.vicary@cl.cam.ac.uk</a>><br>To: Jim Diamond <<a href="mailto:jdiamond@acadiau.ca" target="_blank">jdiamond@acadiau.ca</a>><br>Cc: Ross Moore <<a href="mailto:ozross@icloud.com" target="_blank">ozross@icloud.com</a>>, <a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a><br>Bcc: <br>Date: Fri, 14 Jan 2022 17:31:11 +0000<br>Subject: Re: [pdftex] Caching intermediate compilation results for near-real-time PDF re-renders during editing<br>Hi Jim, Ross and others, thanks for your further comments.<br>
<br>
I certainly agree that some parts of the document have substantially<br>
nonlocal effects, e.g. references, citations, indexing, and similar. I<br>
should have made clear, any functionality that ordinarily requires<br>
more than one pass of pdflatex is completely outside the scope of what<br>
I am suggesting. (This is the behaviour of BaKoMa TeX -- if you want<br>
your equation references etc to update, you have to fire off a full<br>
recompile.)<br>
<br>
> I would guess the development effort to<br>
> do this would be considerable for someone who thoroughly knows the<br>
> internals of the TeX engine, and perhaps a tremendous effort for<br>
> someone starting from scratch.<br>
<br>
I don't know anything about TeX, but I can develop code, and this<br>
feature is sufficiently important to my working style, that I would<br>
potentially be interested to take it on as a project, particularly if<br>
others could be involved.<br>
<br>
> what I<br>
> described may be (far?) less than BaKoMa TeX does anyway; the author<br>
> of that had undoubtedly given it a *lot* more thought than me.<br>
<br>
BaKoMa has its own issues, and is of course no longer being developed.<br>
At some point it will become incompatible with modern latex packages.<br>
<br>
I think it would be great to have this fast-recompile feature in the<br>
open-source world. It doesn't matter if it has limitations at first,<br>
people will improve it over time.<br>
<br>
> If your jobs are not compiling quickly enough for you, then the best option could well be to update your hardware, rather than fiddle with the fundamental design of the software.<br>
<br>
My hardware is very good thank you! :) I of course understand many<br>
people might think this way.<br>
<br>
For 10 years, I have worked on my latex documents with my code on the<br>
left side of my screen, and the BaKoMa preview on the right, giving me<br>
ultra-fast updating at the level of individual keystrokes (as long as<br>
the document is in a syntactically valid state.) These can be large<br>
documents, taking minutes for a full recompilation, and I am often<br>
making minute adjustments to graphical styles, tikz diagrams, etc. I<br>
am **multiple times** more productive with this system than I would be<br>
otherwise. I completely avoid the inconveniences of having to split my<br>
sections and figures into different files, and all those workarounds,<br>
and I must have saved cumulative weeks of my life waiting for<br>
documents to compile.<br>
<br>
Cheers,<br>
Jamie<br>
<br>
<br>
On Fri, Jan 14, 2022 at 5:01 PM Jim Diamond <<a href="mailto:jdiamond@acadiau.ca" target="_blank">jdiamond@acadiau.ca</a>> wrote:<br>
><br>
> Ross,<br>
><br>
> It seems your mail program clobbers the quoting in the plain text<br>
> part.<br>
><br>
> All,<br>
><br>
> At the cost of incorrect quoting below, I'll carry on with the email as-is.<br>
><br>
><br>
> On Fri, Jan 14, 2022 at 14:06 (+1100), Ross Moore wrote:<br>
><br>
> > Hi Jim, Karl and others.<br>
><br>
> > From: Jim Diamond via pdftex <<a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a><mailto:<a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a>>><br>
> > Date: 14 January 2022 at 12:26:27 pm AEDT<br>
> > To: Karl Berry <<a href="mailto:karl@freefriends.org" target="_blank">karl@freefriends.org</a><mailto:<a href="mailto:karl@freefriends.org" target="_blank">karl@freefriends.org</a>>>, <a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a><mailto:<a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a>><br>
> > Subject: Re: [pdftex] Caching intermediate compilation results for near-real-time PDF re-renders during editing<br>
> > Reply-To: Jim Diamond <<a href="mailto:jdiamond@acadiau.ca" target="_blank">jdiamond@acadiau.ca</a><mailto:<a href="mailto:jdiamond@acadiau.ca" target="_blank">jdiamond@acadiau.ca</a>>><br>
><br>
> > Hi all,<br>
><br>
> > On Thu, Jan 13, 2022 at 16:36 (-0700), Karl Berry wrote:<br>
><br>
> > Hi Jamie - thanks for the interesting message. Thanh could say more, but<br>
> > FWIW, here are my reactions (in short, "sounds impossible").<br>
><br>
> > That statement may be true in one very limited sense only.<br>
> > In typical real-world documents, anything that occurs anywhere<br>
> > can have an effect on any other page of the final PDF.<br>
><br>
> This is true. But see below.<br>
><br>
> > I recognize that you are employing hyperbole for effect here. But<br>
> > thinking about the OP's question, I wonder... just how many variables<br>
> > are in play after a shipout?<br>
><br>
> > A shipout of a page does *not* mean that what comes afterwards is like<br>
> > a whole separate stand-alone document.<br>
><br>
> > To process what comes next still relies on everything that has been setup<br>
> > earlier, in terms of how macros will expand, or even what is defined.<br>
> > Think about targets of cross-references, citations, hyperlinks, etc.<br>
><br>
> Good point.<br>
><br>
> > There is no finite set of “variables” whose values can be saved.<br>
><br>
> Surely the collection of registers, macros and other objects<br>
> defining the state of the computation after a shipout is finite.<br>
><br>
> > You would need a snapshot of a portion of the memory,<br>
> > as well as a way to sensibly make use of it.<br>
><br>
> Which gets us back to the OP's question.<br>
><br>
><br>
> > Suppose a small change is then made to the latex source, such that the<br>
> > compiler determines this change would first affect page k.<br>
><br>
> > I can't imagine how that could be determined without retypesetting the<br>
> > entire document.<br>
><br>
> > Agreed.<br>
> > Theoretically, it is like the Halting problem for Turing machines.<br>
> > While the output is sometimes predictable, in general<br>
> > you can only know what a program will do by running it.<br>
> > And will it even stop? ... allowing you to examine the complete output<br>
> > that it will produce? In general, NO.<br>
><br>
> I'd argue that solving the halting problem is sufficient but not<br>
> strictly necessary.<br>
><br>
> > Currently, various editors and document viewers can do lookups and<br>
> > reverse lookups so that one can go from a point in the source doc to<br>
> > the corresponding line in the compiled document, and vice versa.<br>
><br>
> > There is no way to predict what any given change will affect.<br>
><br>
> Perhaps not, reports of the features of BaKoMa TeX (which I have<br>
> never used) notwithstanding.<br>
><br>
> > Is that really true? ***Except for multi-page paragraphs (bah!),<br>
> > tables and similar***, is it possible for a change on page N of a<br>
> > document to affect anything before page N-1?<br>
><br>
> > Absolutely.<br>
> > That’s what the .aux file is typically used for.<br>
> > And many packages have their own auxiliary files which write into a file,<br>
> > so that extra information is available to possibly affect any page N-k<br>
> > (for any value of k) on the next processing run.<br>
><br>
> It is true that if a change to the source causes the .aux (and<br>
> similar) files to be changed, that any/all of the document might be<br>
> changed the next time it is compiled.<br>
><br>
> But should we conclude that we can't do anything useful here,<br>
> following the OP's question? I think we can. (See below.)<br>
><br>
><br>
> > (While I can see change<br>
> > to a word in a continued paragraph on page N changing the typesetting<br>
> > of that paragraph, and thus possibly how page N-1 should be laid out,<br>
> > can page N-2 be affected?)<br>
><br>
> > This kind of thing is simple enough; but still not always.<br>
> > If the editing forces something onto the next page,<br>
> > the overall effect just can get more and more complicated.<br>
><br>
> <snip><br>
><br>
><br>
> > I’ve been a TeX user since the late 1980s.<br>
> > In that time the speed of processing has increased considerably – due mostly<br>
> > to the speed of the (laptop or desktop) computer doing the processing.<br>
><br>
> > Today we do things that would have been inconceivable back last<br>
> > century, precisely because of the greater speed and available<br>
> > memory. This growth is known as Moore’s Law – though not due to me,<br>
> > nor any known relative.<br>
><br>
> <snip><br>
><br>
> > I believe very strongly in the benefits of ultra-fast recompilation<br>
><br>
> > If your jobs are not compiling quickly enough for you, then the best option<br>
> > could well be to update your hardware, rather than fiddle with<br>
> > the fundamental design of the software.<br>
><br>
> I've been a TeX user since the early 1980's. Yes, things have sped up<br>
> considerably. However, to be blunt, I think the suggestion "get<br>
> faster hardware" is a bit on the obnoxious side. While the OP may<br>
> have more money than God, I have heard that there are lots of people<br>
> on the planet with limited financial means, and some of them may have<br>
> to do their computing with a low-end Raspberry Pi or even something<br>
> less capable. (And, IMHO, people buying into the "just get<br>
> more/faster hardware" mantra is why we need 8 GB of RAM to look at a<br>
> web page.)<br>
><br>
><br>
> Anyway, for what it's worth here is a thought of how compilation could<br>
> be sped up to help someone quickly preview their documents.<br>
><br>
> There could be two types of "re-compilation":<br>
> (1) A full (re-)compilation, perhaps running pdf(la)tex the usual<br>
> number of times to ensure all the ToC entries, cross references,<br>
> and so on are done correctly.<br>
> These runs (only the last one is really relevant) could save<br>
> whatever computation state is needed at the end of each page.<br>
> Further, similar to synctex, it could record a correspondence<br>
> between source locations and PDF locations.<br>
> (2) A "best-effort", "fast" re-compilation could look at where in the<br>
> source the first change is since the last "full" (re-)compilation;<br>
> the editor would have to keep track of this. Suppose the point of<br>
> change was found at page N in the most recent full re-compilation.<br>
> Recognizing that this change *might* have affected previous pages,<br>
> it could boldly carry on and load the state of the universe at the<br>
> end of page N-1 and then carry on the compilation from there.<br>
><br>
> This best-effort compilation would allow the user to quickly see how<br>
> local changes look, at the cost of the user needing to recognize that<br>
> a full re-compilation may change things dramatically. But in many<br>
> cases, this might be enough to make a user happy.<br>
><br>
> Jamie: having said all that, I would guess the development effort to<br>
> do this would be considerable for someone who thoroughly knows the<br>
> internals of the TeX engine, and perhaps a tremendous effort for<br>
> someone starting from scratch. I'd further guess that Unless some grad<br>
> student out there finds this to be an interesting project, and his/her<br>
> supervisor thinks it can be turned into a thesis and/or publishable<br>
> material, I'm not sure I see this happening, even though what I<br>
> described may be (far?) less than BaKoMa TeX does anyway; the author<br>
> of that had undoubtedly given it a *lot* more thought than me.<br>
><br>
><br>
> Cheers.<br>
> Jim<br>
<br>
<br>
<br><br><br>---------- Forwarded message ----------<br>From: Doug McKenna <<a href="mailto:doug@mathemaesthetics.com" target="_blank">doug@mathemaesthetics.com</a>><br>To: pdftex <<a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a>><br>Cc: <br>Bcc: <br>Date: Fri, 14 Jan 2022 12:55:05 -0700 (MST)<br>Subject: Re: [pdftex] Caching intermediate compilation results for near-real-time PDF re-renders during editing<br>For documents with tables of contents, extending a section or chapter title by one word (or even letter) on page N could conceivably increase the earlier Table of Contents by one line, which could conceivably cause the table of contents to require one more page, which then could change the page numbering of every page following in the rest of the document.<br>
<br>
This is one reason why Knuth chose fixed-point arithmetic, and TeX's own implementation of fixed-point math, to avoid the butterfly/avalanche effect of even a single low-order bit difference in typesetting measurements.<br>
<br>
Doug McKenna<br>
<br>
<br><br><br>---------- Forwarded message ----------<br>From: Jamie Vicary <<a href="mailto:jamie.vicary@cl.cam.ac.uk" target="_blank">jamie.vicary@cl.cam.ac.uk</a>><br>To: Doug McKenna <<a href="mailto:doug@mathemaesthetics.com" target="_blank">doug@mathemaesthetics.com</a>><br>Cc: pdftex <<a href="mailto:pdftex@tug.org" target="_blank">pdftex@tug.org</a>><br>Bcc: <br>Date: Fri, 14 Jan 2022 20:27:05 +0000<br>Subject: Re: [pdftex] Caching intermediate compilation results for near-real-time PDF re-renders during editing<br>Hi Doug, thanks for your message. Yes I agree. This does not present<br>
an issue, because the proposal is, when doing a "fast recompilation",<br>
not to read in new values from the .aux file. So the table of<br>
contents, equation references, etc, would not update. If the user<br>
wants to update these, they can trigger a separate full compile. This<br>
is how BaKoMa handles it, and it works very well.<br>
<br>
Cheers,<br>
Jamie<br>
<br>
On Fri, Jan 14, 2022 at 7:55 PM Doug McKenna <<a href="mailto:doug@mathemaesthetics.com" target="_blank">doug@mathemaesthetics.com</a>> wrote:<br>
><br>
> For documents with tables of contents, extending a section or chapter title by one word (or even letter) on page N could conceivably increase the earlier Table of Contents by one line, which could conceivably cause the table of contents to require one more page, which then could change the page numbering of every page following in the rest of the document.<br>
><br>
> This is one reason why Knuth chose fixed-point arithmetic, and TeX's own implementation of fixed-point math, to avoid the butterfly/avalanche effect of even a single low-order bit difference in typesetting measurements.<br>
><br>
> Doug McKenna<br>
<br>
<br>
</blockquote></div>