[luatex] Hash tokens meaning

[Arthur Reutenauer]

> No, but they did ask me to enter the pool. I learned by trial and error.

  That's your problem here.  What are you trying to do, and what were
the errors you encountered?  So far as I can see you're not actually
doing anything real, only asking vague questions, the purpose of which
is not clear at all.

> I honestly didn't realize that I asked something "deep" or
> "sensitive", I thought there just was some kind a cheatsheet
> somewhere that I didn't find by myself. I was wrong.

  Not only is there no such document, but if it did it would probably
not help you very much.  The problem with the word "cheatsheet" is that
the information it contains is only valuable to you if you actually know
something beforehand.  It doesn't really help you to cheat, more to
quickly find something that temporarily slipped your mind, or that you
learned a long time ago and need to brush up on.

  Now, being well understood that the answers probably won't help you at
all, there is no reason why we shouldn't try and answer your questions,
to the extent that they make sense:

> I am trying to grasp the meaning of the hash tokens I find in a latex 
> document. I am a novice TeX user, so please bear with my (almost surely) 
> stupid questions: I've only used LaTeX, and never looked into the pit :)

  First of all, I believe that what you're calling "hash token" is a
control sequence, i. e., any sequence of characters starting with
backslash (\).  It's not necessarily always backslash, by the way, but
usually is.  Is that what you mean?

> - First of all, I find many tokens that contain @ or ! or ^, for example 
> "tagsleft at false", "ps at headings", "\T1\^-\i", or "!!stringa": do these 
> characters have a special meaning?

  Not necessarily.  The commercial at sign (@) is often used in LaTeX2e
packages by a special convention, that dictates that its use in control
sequence names is reserved to developers writing packages, and that
users should not type them in their documents.  This enables developers
to be reasonably certain that their internal commands won't be
overwritten by users at typesetting time, and is enforced at a technical
level (by changing @'s category code); but it is no more than a
convention, and it's not uncommon for users to change the category code
themselves at times (the LaTeX kernel even provides commands to switch
back and forth).

  The command \^ takes one argument and puts a circumflex accent over
it.  The caret character is only visual and mnemonic.

  I don't know where you took "!!stringa" from, but it seems to be a
similar case to @, using the exclamation mark to ensure that the control
sequence won't be overwritten by users (as ! is usually not legal in
control sequences).

> - Why do I have so many "blank" tokens? At the end of this you'll find a 
> sample from a dump file viewed from emacs, so an encoding of special 
> characters is visible.

  I don't know what you mean.  In the file you attached, there's only
one blank line, and since you're not telling us how you produced the
file exactly, it could be anything.

> -Are tokens as 
> "<5><6><7><8><9>gen*cmbx<10><10.95>cmbx10<12><14.4><17.28><20.74><24.88>cmbx12" 
> real, or is it some kind of "encoding" that I see?

  They're not real, of course, they're imaginary tokens from the land of
Narnia.  Ah, the Narnian tokens!  Many a night I've spent hunting down
those that, like sheep, had gone astray, and turned them back to the
right path.

  (Or, they could be related to the NFSS.)

> - Are these tokens portable? I mean, a friend of mine on windows will 
> produce the very same tokens with the same document?

  Assuming again that you mean control sequences by "tokens", yes,
they're extremely portable, if they come from the LaTeX kernel.  If
they've been defined by a package, your correspondent needs to have the
same package, of course, but there usually are very few incompatibility
between OSes.  XeTeX, and to some extent LuaTeX, have changed that
partly.

> - All \label{something} seem to be processed as token "r at something". Is 
> this real, or just my imagination?

  See above about Narnia.  Seriously, I have no idea what you mean.  The
vast majority of control sequences are expandable, meaning that their
low-level meaning usually is something else than what you type.  That is
of course the whole point of a macro language; there would be little
advantage to defining commands such as \label, if all "\label{something}"
did was to print "\label{something}" on the page.  It might be that in
the course of being expanded, \label{something} produces a control
sequence \r at something; I do not know the details of how it's implemented,
and have no inclination to look into it.  I fail to see how it's
interesting anyway.

> Again, I am sorry to ask such trivial questions, but I didn't find 
> anything anywhere on the internals of TeX!

  Read the TeXbook.

	Arthur