[tex-k] Does TeX's Input Processor Tokenize The Entire Input File First?

[Tomas Rokicki]

Or Jon.

On Wed, Apr 17, 2019 at 12:41 AM Tomas Rokicki <rokicki at gmail.com> wrote:

> Use the source, Luke.  I mean Joseph.
>
> On Tue, Apr 16, 2019 at 11:13 PM Joseph Wright <
> joseph.wright at morningstar2.co.uk> wrote:
>
>> On 17/04/2019 04:25, Jon Forrest wrote:
>> > I'm trying to understand TeX internals.
>> >
>> > One thing I haven't found a definitive explanation of is whether the
>> > input processor tokenizes the entire input file before the execution
>> > processor gets the token list. TexbyTopic says "... all [processor]
>> > levels are simultaneously  active, and there is interaction between
>> > them.". Given the primitive  computing environment TeX was developed in,
>> > it's hard to imagine how the  4 processors could run simultaneously. As
>> > far as I can see, TeX has no multi-threading ability so it would be up
>> > to TeX to "process" switch.
>> >
>> > So, how and why do the processors run simultaneously? How much of the
>> > input file gets tokenized before the token list is handed to the
>> > expansion processor?
>> >
>> > I wonder if Knuth were writing TeX today would he use multi-threading?
>> >
>> > Cordially,
>> > Jon Forrest
>> >
>>
>> TeX tokenizes one item at a time. That's seen of course in the fact that
>> one can change catcodes during a document
>>
>>      \catcode`~=13 %
>>      \def~{Hello}%
>>      \catcode`~=12 %
>>      ~ world
>>      \bye
>>
>> As such, the different stages of TeX processing cannot be run in parallel.
>>
>> Joseph
>>
>
>
> --
> --  http://cube20.org/  --  http://golly.sf.net/  --
>


-- 
--  http://cube20.org/  --  http://golly.sf.net/  --
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tug.org/pipermail/tex-k/attachments/20190417/6711c46c/attachment.html>