[luatex] Hash tokens meaning

[Arthur Reutenauer]

> I supposed it was clear, because I posted to luatex asking about
> hash tokens. I was obviously misleading others! :)

  You really could have simply mentioned you were using tex.hashtokens()
from the LuaTeX library.  "Hash token" is not part of the standard TeX
vocabulary, but "token" is, with a different meaning.

> That is not my purpose, and by the way, yes I look at the memory
> when trying to figure out how a piece of software works (it's part
> of my job), especially when you assume that you don't have the
> source code.

  OK, but you do have the source code here: why refuse to use it?

> I attached parts of it, since the symbol table is 70K.

  I can't see anything attached, and I meant the TeX source anyway.
That's the input.

> Because it's new. I thought of it as an escaping in C, a sort, let's
> say this, of protection of the next character, as in \% (the same
> way in C for \").

  That's understandable, but TeX's control sequence mechanism is
different from escaping.  A control sequence is anything starting with a
character having category code 0 -- that character is usually \, but can
actually be anything -- and that translates to something else during
macro expansion.  And you can even make \ a part of a control sequence's
name, if you want, but that's a little bit convoluted (and not very
useful).

  And I did leave several parts out, read the TeXbook for the full story.

> Yes, I knew that once opened the hash file with an hex editor. I
> knew TeX didn't have support for unicode, and I thought that
> lualatex translated into TeX, which produced an output. So a unicode
> string was unexpected, and I thought I messed up with my dump code.

  That's another misconception, I'm afraid.  LuaTeX doesn't translate to
TeX, it is an extension of TeX.  It augments the original code base with
many new features -- including, obviously, Lua -- and its input and
output are in UTF-8 (by default).

  (And the code base has been converted from Pascal to C, etc., etc.,
there are many subtopics to that aspect.)

> Pardon me, but I'm used to write code in C, assembly, C++, or
> whatever other programming language (mainly those three, in that
> order). TeX is very, very different.

  Yes, and it is a programming language too (specifically, a macro
language).  It would be useful if you would stop resisting the notion.

>>    Far more interesting are the different commands defined by LaTeX when
>> \label is called, look for "ltxref.dtx" in latex.ltx.  The letter "r"
>> (in \r at something) is introduced in a macro called \newlabel (line 3881
>> of my copy of latex.ltx), and "@" in \@newl at bel, one line above it.
> 
> That is awesome, I now have a place to start!

  I still don't see what's exciting about these few macros, but if it
makes you happy ...

> Anyway, at some point there *is* a static version of a code
> somewhere, otherwise there would be no output.

  I'm not sure that makes sense, but let's just leave it at that for the
time being.

>                                                Yes, I am biased by
> my job and education, but I find hard to grasp the opposition to
> this approach.

  I can't believe I'm reading that.  *You* are the one coming towards
TeX with a huge bias and a host of misconceptions, and you're talking
about opposition to your approach on *our* part?  You freely admit that
you don't understand the way a source file is structured, and you want
us to help you analyse a file that you've produced according to your own
taste, without even showing us what the input was?  You won't tell us
your name, and use a pseudonym that, when you're in the position of a
student, is at best snooty?  If it wasn't for our expertise, your
approach wouldn't have led you anywhere.  Sensei my *ss.

	Arthur