[luatex] Hash tokens meaning
senseiwa at gmail.com
Wed May 29 12:25:15 CEST 2013
On 5/28/13 6:11pm, Arthur Reutenauer wrote:
> You really could have simply mentioned you were using tex.hashtokens()
> from the LuaTeX library. "Hash token" is not part of the standard TeX
> vocabulary, but "token" is, with a different meaning.
As I said, I didn't post to TeX but to Lua because I used LuaLaTeX, and
the subject of this post was about Hash tokens. I thought it was a
LuaTeX feature, not a TeX one (no "\" in any token, so I didn't
recognize TeX except for the names which are the same, except for "\"),
and I thought it was clear. So, again as I said, I was wrong, I wasn't
> OK, but you do have the source code here: why refuse to use it?
Where did I ever refuse to use it? I said I would get myself the code
and the TeX book.
>> Because it's new. I thought of it as an escaping in C, a sort, let's
>> say this, of protection of the next character, as in \% (the same
>> way in C for \").
> That's understandable, but TeX's control sequence mechanism is
> different from escaping. A control sequence is anything starting with a
> character having category code 0 -- that character is usually \, but can
> actually be anything -- and that translates to something else during
> macro expansion. And you can even make \ a part of a control sequence's
> name, if you want, but that's a little bit convoluted (and not very
I understand now.
> And I did leave several parts out, read the TeXbook for the full story.
Yes, I am reading it.
>> Yes, I knew that once opened the hash file with an hex editor. I
>> knew TeX didn't have support for unicode, and I thought that
>> lualatex translated into TeX, which produced an output. So a unicode
>> string was unexpected, and I thought I messed up with my dump code.
> That's another misconception, I'm afraid. LuaTeX doesn't translate to
> TeX, it is an extension of TeX. It augments the original code base with
> many new features -- including, obviously, Lua -- and its input and
> output are in UTF-8 (by default).
> (And the code base has been converted from Pascal to C, etc., etc.,
> there are many subtopics to that aspect.)
A common misconception that I've fallen into.
>> Pardon me, but I'm used to write code in C, assembly, C++, or
>> whatever other programming language (mainly those three, in that
>> order). TeX is very, very different.
> Yes, and it is a programming language too (specifically, a macro
> language). It would be useful if you would stop resisting the notion.
Where did I resist? I accept the notion, and I am just trying to explain
*WHY* I thought otherwise. It's not that I am saying that I was right.
>>> Far more interesting are the different commands defined by LaTeX when
>>> \label is called, look for "ltxref.dtx" in latex.ltx. The letter "r"
>>> (in \r at something) is introduced in a macro called \newlabel (line 3881
>>> of my copy of latex.ltx), and "@" in \@newl at bel, one line above it.
>> That is awesome, I now have a place to start!
> I still don't see what's exciting about these few macros, but if it
> makes you happy ...
I start walking, then I run. Since I don't know for instance where the
equivalent of "main()" is in the LaTeX code, you provided me a
>> Anyway, at some point there *is* a static version of a code
>> somewhere, otherwise there would be no output.
> I'm not sure that makes sense, but let's just leave it at that for the
> time being.
There is a static version because somewhere the codes are in RAM,
somewhere the final version of a LaTeX command can be found.
>> Yes, I am biased by
>> my job and education, but I find hard to grasp the opposition to
>> this approach.
> I can't believe I'm reading that. *You* are the one coming towards
> TeX with a huge bias and a host of misconceptions, and you're talking
> about opposition to your approach on *our* part? You freely admit that
> you don't understand the way a source file is structured, and you want
> us to help you analyse a file that you've produced according to your own
> taste, without even showing us what the input was? You won't tell us
> your name, and use a pseudonym that, when you're in the position of a
> student, is at best snooty? If it wasn't for our expertise, your
> approach wouldn't have led you anywhere.
I never asked to analyze a file, reread my post. I asked about specific
I am quite sure of my knowledge to recognize that I am not able to make
it on my own.
So, since I recognize my bias, I asked for something that in the
programming language community is clear. If I have a function in C
called "main()" how will this be translated? Well, you'll end up with a
symbol "_main". If you don't see why this connection makes sense... but
anyway, if you read my posts with calm and zen, you will see that I
always said "I was wrong".
Should I flog myself for being wrong? It's the only way to improve.
Again, I never had an attitude of someone saying "you pricks don't know
s*it and I do LOL". If you ever see in one of my posts that, I didn't do
> Sensei my *ss.
And that's very mature.
I understand you know everything on every topic, and don't need to ask
questions to those who know more than you. Well, I am sorry I am not
skilled on these matters, that's why I asked.
A "RTFM" attitude won't make you any more respected.
More information about the luatex