# [tex-k] Does TeX's Input Processor Tokenize The Entire Input File First?

Joseph Wright joseph.wright at morningstar2.co.uk
Wed Apr 17 08:12:55 CEST 2019

On 17/04/2019 04:25, Jon Forrest wrote:
> I'm trying to understand TeX internals.
>
> One thing I haven't found a definitive explanation of is whether the
> input processor tokenizes the entire input file before the execution
> processor gets the token list. TexbyTopic says "... all [processor]
> levels are simultaneously  active, and there is interaction between
> them.". Given the primitive  computing environment TeX was developed in,
> it's hard to imagine how the  4 processors could run simultaneously. As
> far as I can see, TeX has no multi-threading ability so it would be up
> to TeX to "process" switch.
>
> So, how and why do the processors run simultaneously? How much of the
> input file gets tokenized before the token list is handed to the
> expansion processor?
>
> I wonder if Knuth were writing TeX today would he use multi-threading?
>
> Cordially,
> Jon Forrest
>

TeX tokenizes one item at a time. That's seen of course in the fact that
one can change catcodes during a document

\catcode~=13 %
\def~{Hello}%
\catcode~=12 %
~ world
\bye

As such, the different stages of TeX processing cannot be run in parallel.

Joseph