[luatex] Allowing or switching to string indexes in Lua bytecode registers

[David Carlisle]

On 5 September 2015 at 17:42, Kalrish Bäakjen <kalrish.baakjen at gmail.com> wrote:



> Thank you very much!
>
> I use TeX Live and, unfortunately, there doesn't seem to be a way to
> use SVN LaTeX here. As I think ltluatex.dtx[1] is the relevant file, I
> have tried to locate it in my tree, but haven't found it. I haven't
> been able to use the Lua function (luatexbase.new_bytecode, as it
> appears in ltluatex.dtx[1]) either; could you please tell me which
> package must I load?

The relevant files are not yet on ctan (and so not in texlive) the sources
are available from the web view of the svn, but you would need to
extract the ltluatex.tex and ltluatex.lua files or wait for the next release
(hopefully in a few weeks)

meanwhile if you have ltluatex.dtx from SVN (make sure you have the current one,
it's been updated a few times today)  then just make a file
ltluatex.ins that looks like

\input docstrip
\generate{\file{ltluatex.tex}{\from{ltluatex.dtx}{tex,plain}}}
\nopostamble
\nopreamble
\generate{\file{ltluatex.lua}{\from{ltluatex.dtx}{lua}}}

run tex in that to get ltluatex.tex and .lua then

\input{ltluatex}

in a document should work.


>
> In the meantime, I've been scratching my head. My idea, for packages
> that use Lua code, is as follows:
> - Regardless of whether or not we are on iniLuaTeX, we must get a
> loader function for our Lua code through loadfile:
>   local loader_function = assert( loadfile('mycode.lua') )
> - If we are on iniLuaTeX, we must:
>  1. ask the bytecode allocator (\newluabytecode) for a number (the
> index), which we store in \myindex and which, I believe, would be
> dumped in the format:
>   \newluabytecode\myindex
>  2. save our Lua loader function, which we got by loadfile, in the
> slot referenced by that index:
>   \directlua{ lua.bytecode[\myindex] = loader_function }
>  3. specify an "everyjob" via \everyjob that restores our Lua code by
> calling the loader function stored at the bytecode slot referenced by
> the TeX index:
>   \everyjob\expandafter{ \the\everyjob\directlua{ lua.bytecode[\myindex]() } }
> - If we are on a regular run, we just have to run the loader function
> to actually load our Lua code and be able to use it:
>   \directlua{ loader_function() }

Yes, something like that:-)
>
> This logic, however, doesn't seem to play well with Lua's require
> (and, by extension, luatexbase-modutils' require_module, which uses it
> under the hood) function[2], because it loads the Lua code internally;

in ltluatex the assumption is that you can just use require() rather than
needing a special wrapper (as require uses kpse anyway now)
the luatexbase emulation package does define \RequireLuaModule

> packages doing \dofile{ require('mycode.lua') } have no way of
> accessing the loader function, which is what needs to be stored in
> bytecode registers.

well yes that's true (suggestions welcome) but it seems to me that you can't
byte compile arbitrary code, and this is such a case, that  (for
simple use at least)
the code could be structured so that you can  byte compile each file separately.


On the other hand a more general scheme could probably work although
I'd need to try building a test case to follow the details below.
I see the general direction you are suggesting but some of the details
escape me:-)

>  Perhaps a custom searcher (see [3]) could be
> introduced. This searcher would be based on the default second
> searcher. First, it would load the code from the resolved path:
>   -- resolved_path is the path of the file containing the Lua code;
> this path would be obtained with package.searchpath
>   local loader_function = assert( loadfile(resolved_path) )
> This special searcher would have to know if it's on a dumping session
> or a normal run.
> If on a normal run, it would just execute that loader function to
> actually load the code and be done with it:
>   loader_function()
> If on iniLuaTeX, it would get a bytecode register index and save the
> loader function in the corresponding slot. I haven't yet figured out,
> however, how would this special searcher of mine keep track of the
> bytecode register index that it had been assigned by the allocation
> manager. Perhaps a LuaTeX attribute? Like this:
>   -- Get an index by the bytecode register allocator
>   local register_index = luatexbase.new_bytecode(resolved_path)
>   -- Store it in a LuaTeX attribute named after the resolved path of
> the Lua file we loaded with loadfile. These attributes are dumped, I
> think
>   lua.attribute[resolved_path] = register_index
>   -- Finally, spit out an "everyjob" to the TeX engine. This
> "everyjob" is supposed to execute the loader function later, during
> the normal run. We use the resolved path as the name of our attribute
>   tex.sprint( [[\everyjob\expandafter{ \the\everyjob\directlua{
> lua.bytecode[\csname ]] , resolved_path , [[\endcsname]() } }]] )
>
> This may be getting too awful, but I think that the possibility that
> Lua-using packages are dumped (along with their Lua code) is important
> for the future, as Lua code can be noticeably slower to process than
> "good old" TeX code.
>

yes we'll see what we can do, but also this would be the first latex format with
built in support for luatex allocations, so we need to be reasonably
conservative
It should however form the basis for building more advanced interfaces as above.

David