[latex3-commits] [git/LaTeX3-latex3-latex3] luaintarray: Lua based intarrays (96d85df9b)

Marcel Fabian Krüger tex at 2krueger.de
Sun Nov 7 06:30:17 CET 2021


Repository : https://github.com/latex3/latex3
On branch  : luaintarray
Link       : https://github.com/latex3/latex3/commit/96d85df9b5bdb6c52693116ac8e3f02b7096770f

>---------------------------------------------------------------

commit 96d85df9b5bdb6c52693116ac8e3f02b7096770f
Author: Marcel Fabian Krüger <tex at 2krueger.de>
Date:   Mon Aug 10 21:40:33 2020 +0200

    Lua based intarrays


>---------------------------------------------------------------

96d85df9b5bdb6c52693116ac8e3f02b7096770f
 l3kernel/l3.ins                                    |   3 +-
 l3kernel/l3bootstrap.dtx                           |   1 +
 l3kernel/l3intarray.dtx                            | 528 ++++++++++++++++++++-
 l3kernel/l3luatex.dtx                              |  78 +++
 l3kernel/testfiles/m3intarray001.luatex.tlg        |  63 +--
 l3kernel/testfiles/m3intarray001.lvt               |   4 +-
 .../{m3skip001.tlg => m3skip001.luatex.tlg}        |  18 +-
 .../{m3skip005.tlg => m3skip005.luatex.tlg}        |  22 +-
 8 files changed, 633 insertions(+), 84 deletions(-)

diff --git a/l3kernel/l3.ins b/l3kernel/l3.ins
index 34a46aaed..265fde8ac 100644
--- a/l3kernel/l3.ins
+++ b/l3kernel/l3.ins
@@ -78,7 +78,7 @@ and all files in that bundle must be distributed together.
         \from{l3file.dtx}       {package}
         \from{l3skip.dtx}       {package}
         \from{l3keys.dtx}       {package}
-        \from{l3intarray.dtx}   {package}
+        \from{l3intarray.dtx}   {package,tex}
         \from{l3fp.dtx}         {package}
         \from{l3fp-aux.dtx}     {package}
         \from{l3fp-traps.dtx}   {package}
@@ -171,6 +171,7 @@ and all files in that bundle must be distributed together.
   \from{l3names.dtx}{package,lua}
   \from{l3sys.dtx}{package,lua}
   \from{l3token.dtx}{package,lua}
+  \from{l3intarray.dtx}{package,lua}
 }}
 
 \endbatchfile
diff --git a/l3kernel/l3bootstrap.dtx b/l3kernel/l3bootstrap.dtx
index c0c676dd9..1f17916bf 100644
--- a/l3kernel/l3bootstrap.dtx
+++ b/l3kernel/l3bootstrap.dtx
@@ -201,6 +201,7 @@
       \expandafter\noexpand\csname prg_return_true:\endcsname
       \expandafter\noexpand\csname prg_return_false:\endcsname
     }\endgroup
+    \ifdefined\newluabytecode\newluabytecode\expl at luadata@bytecode\fi
     \directlua{require("expl3")}%
 %    \end{macrocode}
 %   As the user might be making a custom format, no assumption is made about
diff --git a/l3kernel/l3intarray.dtx b/l3kernel/l3intarray.dtx
index 13ba3a394..d02a6cf19 100644
--- a/l3kernel/l3intarray.dtx
+++ b/l3kernel/l3intarray.dtx
@@ -173,7 +173,525 @@
 %<@@=intarray>
 %    \end{macrocode}
 %
-% \subsection{Allocating arrays}
+% There are two implementations for this module: One \cs{fontdimen} based one
+% for more traditional \TeX\ engines and a Lua based one for engines with Lua support.
+%
+% \subsection{Lua implementation}
+% First, let's look at the Lua variant:
+%
+% We select the Lua version if the Lua helpers were defined. This can be detected by
+% the presence of \cs{@@_gset_count:Nw}.
+%
+%    \begin{macrocode}
+%<*tex>
+\cs_if_exist:NTF \@@_gset_count:Nw
+{
+%</tex>
+%    \end{macrocode}
+%
+% \subsubsection{Allocating arrays}
+%
+% \begin{variable}{\l_@@_loop_int}
+%   A loop index.
+%    \begin{macrocode}
+%<*tex>
+\int_new:N \l_@@_loop_int
+%    \end{macrocode}
+% \end{variable}
+%
+% \begin{variable}{\g_@@_table_int, \l_@@_bad_index_int}
+%   Used to differentiate intarrays in Lua and to record an invalid index.
+%    \begin{macrocode}
+\int_new:N \g_@@_table_int
+\int_new:N \l_@@_bad_index_int
+%    \end{macrocode}
+% \end{variable}
+%
+%    \begin{macrocode}
+\msg_new:nnn { kernel } { negative-array-size }
+  { Size~of~array~may~not~be~negative:~#1 }
+%    \end{macrocode}
+%
+% \begin{macro}{\s_@@}
+%   Used as marker for intarrays in Lua. Followed by an unbraced number
+%   identifying the array and a single space. This format is used to make it
+%   easy to scan from Lua.
+%    \begin{macrocode}
+%</tex>
+%<*lua>
+luacmd('s_@@', function()
+  scan_int()
+  tex.error'Isolated intarray ignored'
+end, 'protected', 'global')
+%</lua>
+%<*tex>
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}{\intarray_new:Nn, \intarray_new:cn}
+% \begin{macro}{\@@_new:N}
+%   Declare |#1| as a tokenlist with the scanmark and a unique number.
+%   Pass the array's size to the Lua helper.
+%   Every \texttt{intarray} must be global; it's enough to run this
+%   check in \cs{intarray_new:Nn}.
+%    \begin{macrocode}
+\cs_new_protected:Npn \@@_new:N #1
+  {
+    \__kernel_chk_if_free_cs:N #1
+    \int_gincr:N \g_@@_table_int
+    \cs_gset:Npx #1 { \s_@@ \int_use:N \g_@@_table_int \c_space_tl }
+  }
+\cs_new_protected:Npn \intarray_new:Nn #1#2
+  {
+    \@@_new:N #1
+    \@@_gset_count:Nw #1 \int_eval:n {#2} \scan_stop:
+    \int_compare:nNnT { \intarray_count:N #1 } < 0
+      {
+        \msg_error:nnx { kernel } { negative-array-size }
+          { \intarray_count:N #1 }
+      }
+  }
+\cs_generate_variant:Nn \intarray_new:Nn { c }
+%</tex>
+%    \end{macrocode}
+% \end{macro}
+% \end{macro}
+%
+% \begin{macro}{@@_table}
+%   Internal helper to scan an intarray token, extract the associated
+%   Lua table and return an error if the input is invalid.
+%    \begin{macrocode}
+%<*lua>
+
+local scan_token = token.scan_token
+local scan_toks = token.scan_toks
+local get_csname = token.get_csname
+local put_next = token.put_next
+local s_@@ = token.create's_@@'
+local use_none
+
+local use_i = token.create'use:n'
+if use_i.cmdname == "undefined_cs" then
+  use_i = nil
+else
+  use_none = token.create'use_none:n'
+end
+
+local scan_argument = token.scan_argument
+local @@_table do
+  local tables = get_luadata and get_luadata'@@' or {[0] = {}}
+  function @@_table()
+    local t = scan_token()
+    if t ~= s_@@ then
+      token.put_next(t)
+      tex.error'intarray expected'
+      return tables[0]
+    end
+    local i = scan_int()
+    local table = tables[i]
+    if table then return table end
+    table = {}
+    tables[i] = table
+    return table
+  end
+  if register_luadata then
+    register_luadata('@@', function()
+      local t = "{[0]={},"
+      for i=1, #tables do
+        t = string.format("%s{%s},", t, table.concat(tables[i], ','))
+      end
+      return t .. "}"
+    end)
+  end
+end
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}[EXP]{\intarray_count:N, \intarray_count:c}
+% \begin{macro}[EXP]{\@@_gset_count:Nw}
+%   Set and get the size of an array. ``Setting the size'' means in this context that
+%   we add zeros until we reach the desired size.
+%    \begin{macrocode}
+
+local sprint = tex.sprint
+
+luacmd('@@_gset_count:Nw', function()
+  if not use_i then
+    use_i = token.create'use:n'
+    use_none = token.create'use_none:n'
+  end
+  local t = @@_table()
+  local n = scan_int()
+  for i=#t+1, n do t[i] = 0 end
+end, 'protected', 'global')
+
+luacmd('intarray_count:N', function()
+  sprint(-2, #@@_table())
+end, 'global')
+%</lua>
+%    \end{macrocode}
+%
+%    \begin{macrocode}
+%<*tex>
+\cs_generate_variant:Nn \intarray_count:N { c }
+%</tex>
+%    \end{macrocode}
+% \end{macro}
+% \end{macro}
+%
+% \subsubsection{Array items}
+%
+% \begin{macro}{\@@_gset:wNwF, \@@_gset:wNw}
+%   The setter provided by Lua. The argument order somewhat emulates the |\fontdimen|
+%   This has been chosen over a more conventional order to provide a delimiter for the numbers.
+%    \begin{macrocode}
+%<*lua>
+luacmd('@@_gset:wNwF', function()
+  local i = scan_int()
+  local t = @@_table()
+  if t[i] then
+    t[i] = scan_int()
+    put_next(use_none)
+  else
+    tex.l_@@_bad_index_int = i
+    scan_int()
+    put_next(use_i)
+  end
+end, 'protected', 'global')
+
+luacmd('@@_gset:wNw', function()
+  local i = scan_int()
+  local t = @@_table()
+  if t[i] then
+    t[i] = scan_int()
+    put_next(use_none)
+  else
+    scan_int()
+    put_next(use_i)
+  end
+end, 'protected', 'global')
+%</lua>
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}{\intarray_gset:Nnn, \intarray_gset:cnn, \__kernel_intarray_gset:Nnn}
+%   Set the appropriate \tn{fontdimen}.  The
+%   \cs{__kernel_intarray_gset:Nnn} function does not use
+%   \cs{int_eval:n}, namely its arguments must be suitable for
+%   \cs{int_value:w}.  The user version checks the position and value
+%   are within bounds.
+%    \begin{macrocode}
+%<*tex>
+\cs_new_protected:Npn \__kernel_intarray_gset:Nnn #1#2#3
+{ \@@_gset:wNw #2 #1 #3 \scan_stop: }
+\cs_new_protected:Npn \intarray_gset:Nnn #1#2#3
+  {
+    \@@_gset:wNwF \int_eval:n {#2} #1 \int_eval:n{#3}
+      {
+        \msg_error:nnxxx { kernel } { out-of-bounds }
+          { \token_to_str:N #1 } { \int_use:N \l_@@_bad_index_int } { \intarray_count:N #1 }
+      }
+  }
+\cs_generate_variant:Nn \intarray_gset:Nnn { c }
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}{\intarray_gzero:N, \intarray_gzero:c}
+%   Set the appropriate \tn{fontdimen} to zero.  No bound checking
+%   needed.  The \cs{prg_replicate:nn} possibly uses quite a lot of
+%   memory, but this is somewhat comparable to the size of the array,
+%   and it is much faster than an \cs{int_step_inline:nn} loop.
+%    \begin{macrocode}
+\cs_new_protected:Npn \intarray_gzero:N #1
+  {
+    \int_zero:N \l_@@_loop_int
+    \prg_replicate:nn { \intarray_count:N #1 }
+      {
+        \int_incr:N \l_@@_loop_int
+        \@@_gset:wNwF \l_@@_loop_int #1 \c_zero_dim {}
+      }
+  }
+\cs_generate_variant:Nn \intarray_gzero:N { c }
+%</tex>
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}[EXP]{\intarray_item:Nn, \intarray_item:cn, \__kernel_intarray_item:Nn}
+% \begin{macro}{\@@_item:wNF,\@@_item:wN}
+%   Get the appropriate entry and perform bound checks.  The
+%   \cs{__kernel_intarray_item:Nn} function omits bound checks and omits
+%   \cs{int_eval:n}, namely its argument must be a \TeX{} integer
+%   suitable for \cs{int_value:w}.
+%    \begin{macrocode}
+%<*lua>
+luacmd('@@_item:wNF', function()
+  local i = scan_int()
+  local t = @@_table()
+  local item = t[i]
+  if item then
+    put_next(use_none)
+  else
+    tex.l_@@_bad_index_int = i
+    put_next(use_i)
+  end
+  sprint(-2, item or 0)
+end, 'global')
+
+luacmd('@@_item:wN', function()
+  local i = scan_int()
+  local t = @@_table()
+  sprint(-2, t[i])
+end, 'global')
+%</lua>
+%    \end{macrocode}
+%
+%    \begin{macrocode}
+%<*tex>
+\cs_new:Npn \__kernel_intarray_item:Nn #1#2
+  { \@@_item:wN #2 #1 }
+\cs_new:Npn \intarray_item:Nn #1#2
+  {
+    \@@_item:wNF \int_eval:n {#2} #1
+      {
+        \msg_expandable_error:nnfff { kernel } { out-of-bounds }
+          { \token_to_str:N #1 } { \int_use:N \l_@@_bad_index_int } { \intarray_count:N #1 }
+      }
+  }
+\cs_generate_variant:Nn \intarray_item:Nn { c }
+%    \end{macrocode}
+% \end{macro}
+% \end{macro}
+%
+% \begin{macro}{\intarray_rand_item:N, \intarray_rand_item:c}
+%   Importantly, \cs{intarray_item:Nn} only evaluates its argument once.
+%    \begin{macrocode}
+\cs_new:Npn \intarray_rand_item:N #1
+  { \intarray_item:Nn #1 { \int_rand:n { \intarray_count:N #1 } } }
+\cs_generate_variant:Nn \intarray_rand_item:N { c }
+%    \end{macrocode}
+% \end{macro}
+%
+% \subsubsection{Working with contents of integer arrays}
+%
+% \begin{macro}{\intarray_const_from_clist:Nn, \intarray_const_from_clist:cn}
+%   The current Lua helpers do not allow to avoid bound checking, so we have
+%   to count the elements first. Otherwise it is basically the same as
+%   \cs{intarray_new:Nn} (which we don't use because it would complain about the
+%   wrong qualifier in debugging mode)
+%   This is not implemented in Lua to ensure that the clist parsing is consistent
+%   with the clist module.
+%    \begin{macrocode}
+\cs_new_protected:Npn \intarray_const_from_clist:Nn #1#2
+  {
+    \@@_new:N #1
+    \@@_gset_count:Nw #1 \clist_count:n {#2} \scan_stop:
+    \int_zero:N \l_@@_loop_int
+    \clist_map_inline:nn {#2}
+      {
+        \int_incr:N \l_@@_loop_int
+        \__kernel_intarray_gset:Nnn #1 \l_@@_loop_int { \int_eval:n {##1} } }
+  }
+\cs_generate_variant:Nn \intarray_const_from_clist:Nn { c }
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}[rEXP]{\intarray_to_clist:N, \intarray_to_clist:c}
+% \begin{macro}[rEXP]{\@@_to_clist:Nn, \@@_to_clist:w}
+%   Loop through the array, putting a comma before each item.  Remove
+%   the leading comma with |f|-expansion.  We also use the auxiliary in
+%   \cs{intarray_show:N} with argument comma, space.
+%    \begin{macrocode}
+\cs_new:Npn \intarray_to_clist:N #1 { \@@_to_clist:Nn #1 { , } }
+\cs_generate_variant:Nn \intarray_to_clist:N { c }
+\cs_new:Npn \@@_to_clist:Nn #1#2
+  {
+    \int_compare:nNnF { \intarray_count:N #1 } = \c_zero_int
+      {
+        \exp_last_unbraced:Nf \use_none:n
+          { \@@_to_clist:w 1 ; #1 {#2} \prg_break_point: }
+      }
+  }
+\cs_new:Npn \@@_to_clist:w #1 ; #2#3
+  {
+    \if_int_compare:w #1 > \intarray_count:N #2 ~
+      \prg_break:n
+    \fi:
+    #3 \__kernel_intarray_item:Nn #2 {#1}
+    \exp_after:wN \@@_to_clist:w
+    \int_value:w \int_eval:w #1 + \c_one_int ; #2 {#3}
+  }
+%    \end{macrocode}
+% \end{macro}
+% \end{macro}
+%
+% \begin{macro}[rEXP]{\__kernel_intarray_range_to_clist:Nnn, \@@_range_to_clist:ww}
+%   Loop through part of the array.
+%    \begin{macrocode}
+\cs_new:Npn \__kernel_intarray_range_to_clist:Nnn #1#2#3
+  {
+    \exp_last_unbraced:Nf \use_none:n
+      {
+        \exp_after:wN \@@_range_to_clist:ww
+        \int_value:w \int_eval:w #2 \exp_after:wN ;
+        \int_value:w \int_eval:w #3 ;
+        #1 \prg_break_point:
+      }
+  }
+\cs_new:Npn \@@_range_to_clist:ww #1 ; #2 ; #3
+  {
+    \if_int_compare:w #1 > #2 \exp_stop_f:
+      \prg_break:n
+    \fi:
+    , \__kernel_intarray_item:Nn #3 {#1}
+    \exp_after:wN \@@_range_to_clist:ww
+    \int_value:w \int_eval:w #1 + \c_one_int ; #2 ; #3
+  }
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}{\__kernel_intarray_gset_range_from_clist:Nnn, \@@_gset_range:Nw}
+%   Loop through part of the array.
+%    \begin{macrocode}
+\cs_new_protected:Npn \__kernel_intarray_gset_range_from_clist:Nnn #1#2#3
+  {
+    \int_set:Nn \l_@@_loop_int {#2}
+    \@@_gset_range:Nw #1 #3 , , \prg_break_point:
+  }
+\cs_new_protected:Npn \@@_gset_range:Nw #1 #2 ,
+  {
+    \if_catcode:w \scan_stop: \tl_to_str:n {#2} \scan_stop:
+      \prg_break:n
+    \fi:
+    \__kernel_intarray_gset:Nnn #1 \l_@@_loop_int {#2}
+    \int_incr:N \l_@@_loop_int
+    \@@_gset_range:Nw #1
+  }
+%    \end{macrocode}
+% \end{macro}
+%
+% \begin{macro}{\intarray_show:N, \intarray_show:c, \intarray_log:N, \intarray_log:c}
+%   Convert the list to a comma list (with spaces after each comma)
+%    \begin{macrocode}
+\cs_new_protected:Npn \intarray_show:N { \@@_show:NN \msg_show:nnxxxx }
+\cs_generate_variant:Nn \intarray_show:N { c }
+\cs_new_protected:Npn \intarray_log:N { \@@_show:NN \msg_log:nnxxxx }
+\cs_generate_variant:Nn \intarray_log:N { c }
+\cs_new_protected:Npn \@@_show:NN #1#2
+  {
+    \__kernel_chk_defined:NT #2
+      {
+        #1 { intarray } { show }
+          { \token_to_str:N #2 }
+          { \intarray_count:N #2 }
+          { >~ \@@_to_clist:Nn #2 { , ~ } }
+          { }
+      }
+  }
+%    \end{macrocode}
+% \end{macro}
+%
+% \subsubsection{Random arrays}
+%
+% \begin{macro}{\intarray_gset_rand:Nn, \intarray_gset_rand:cn}
+% \begin{macro}{\intarray_gset_rand:Nnn, \intarray_gset_rand:cnn}
+% \begin{macro}
+%   {
+%     \@@_gset_rand:Nnn,
+%     \@@_gset_rand:Nff,
+%     \@@_gset_rand_auxi:Nnnn,
+%     \@@_gset_rand_auxii:Nnnn,
+%     \@@_gset_rand_auxiii:Nnnn,
+%     \@@_gset_all_same:Nn,
+%   }
+%   We only perform the bounds checks once.  This is done by two
+%   \cs{@@_gset_overflow_test:nw}, with an appropriate empty argument to
+%   avoid a spurious \enquote{at position \texttt{\#1}} part in the
+%   error message.  Then calculate the number of choices: this is at
+%   most $(2^{30}-1)-(-(2^{30}-1))+1=2^{31}-1$, which just barely does
+%   not overflow.  For small ranges use \cs{__kernel_randint:n} (making
+%   sure to subtract~$1$ \emph{before} adding the random number to the
+%   \meta{min}, to avoid overflow when \meta{min} or \meta{max} are
+%   $\pm\cs{c_max_int}$), otherwise \cs{__kernel_randint:nn}.  Finally,
+%   if there are no random numbers do not define any of the auxiliaries.
+%    \begin{macrocode}
+\cs_new_protected:Npn \intarray_gset_rand:Nn #1
+  { \intarray_gset_rand:Nnn #1 { 1 } }
+\cs_generate_variant:Nn \intarray_gset_rand:Nn { c }
+\sys_if_rand_exist:TF
+  {
+    \cs_new_protected:Npn \intarray_gset_rand:Nnn #1#2#3
+      {
+        \@@_gset_rand:Nff #1
+          { \int_eval:n {#2} } { \int_eval:n {#3} }
+      }
+    \cs_new_protected:Npn \@@_gset_rand:Nnn #1#2#3
+      {
+        \int_compare:nNnTF {#2} > {#3}
+          {
+            \msg_expandable_error:nnnn
+              { kernel } { randint-backward-range } {#2} {#3}
+            \@@_gset_rand:Nnn #1 {#3} {#2}
+          }
+          {
+            \@@_gset_rand_auxi:Nnnn #1 { } {#2} {#3}
+          }
+      }
+    \cs_generate_variant:Nn \@@_gset_rand:Nnn { Nff }
+    \cs_new_protected:Npn \@@_gset_rand_auxi:Nnnn #1#2#3#4
+      {
+        \@@_gset_rand_auxii:Nnnn #1 { } {#4} {#3}
+      }
+    \cs_new_protected:Npn \@@_gset_rand_auxii:Nnnn #1#2#3#4
+      {
+        \exp_args:NNf \@@_gset_rand_auxiii:Nnnn #1
+          { \int_eval:n { #3 - #4 + 1 } } {#4} {#3}
+      }
+    \cs_new_protected:Npn \@@_gset_rand_auxiii:Nnnn #1#2#3#4
+      {
+        \exp_args:NNf \@@_gset_all_same:Nn #1
+          {
+            \int_compare:nNnTF {#2} > \c__kernel_randint_max_int
+              {
+                \exp_stop_f:
+                \int_eval:n { \__kernel_randint:nn {#3} {#4} }
+              }
+              {
+                \exp_stop_f:
+                \int_eval:n { \__kernel_randint:n {#2} - 1 + #3 }
+              }
+          }
+      }
+    \cs_new_protected:Npn \@@_gset_all_same:Nn #1#2
+      {
+        \int_zero:N \l_@@_loop_int
+        \prg_replicate:nn { \intarray_count:N #1 }
+          {
+            \int_incr:N \l_@@_loop_int
+            \__kernel_intarray_gset:Nnn #1 \l_@@_loop_int {#2}
+          }
+      }
+  }
+  {
+    \cs_new_protected:Npn \intarray_gset_rand:Nnn #1#2#3
+      {
+        \msg_error:nnn { kernel } { fp-no-random }
+          { \intarray_gset_rand:Nnn #1 {#2} {#3} }
+      }
+  }
+\cs_generate_variant:Nn \intarray_gset_rand:Nnn { c }
+%    \end{macrocode}
+% \end{macro}
+% \end{macro}
+% \end{macro}
+%
+% \subsection{Font dimension based implementation}
+%
+% Go to the false branch of the conditional above.
+%    \begin{macrocode}
+}{
+%    \end{macrocode}
+%
+% \subsubsection{Allocating arrays}
 %
 % \begin{macro}{\@@_entry:w, \@@_count:w}
 %   We use these primitives quite a lot in this module.
@@ -255,7 +773,7 @@
 %    \end{macrocode}
 % \end{macro}
 %
-% \subsection{Array items}
+% \subsubsection{Array items}
 %
 % \begin{macro}[EXP]{\@@_signed_max_dim:n}
 %   Used when an item to be stored is larger than \cs{c_max_dim} in
@@ -400,7 +918,7 @@
 %    \end{macrocode}
 % \end{macro}
 %
-% \subsection{Working with contents of integer arrays}
+% \subsubsection{Working with contents of integer arrays}
 %
 % \begin{macro}{\intarray_const_from_clist:Nn, \intarray_const_from_clist:cn}
 % \begin{macro}{\@@_const_from_clist:nN}
@@ -527,7 +1045,7 @@
 %    \end{macrocode}
 % \end{macro}
 %
-% \subsection{Random arrays}
+% \subsubsection{Random arrays}
 %
 % \begin{macro}{\intarray_gset_rand:Nn, \intarray_gset_rand:cn}
 % \begin{macro}{\intarray_gset_rand:Nnn, \intarray_gset_rand:cnn}
@@ -624,6 +1142,8 @@
 % \end{macro}
 %
 %    \begin{macrocode}
+}
+%</tex>
 %</package>
 %    \end{macrocode}
 %
diff --git a/l3kernel/l3luatex.dtx b/l3kernel/l3luatex.dtx
index f95ddecc8..71816edff 100644
--- a/l3kernel/l3luatex.dtx
+++ b/l3kernel/l3luatex.dtx
@@ -677,6 +677,84 @@ end
 %    \end{macrocode}
 % \end{macro}
 %
+% \subsection{Preserving iniTeX Lua data for runs}
+%
+%    \begin{macrocode}
+%<@@=lua>
+%    \end{macrocode}
+%
+% The Lua state is not dumped when a forat is written, therefore any Lua
+% variables filled doing format building need to be restored in order to
+% be accessible during normal runs.
+%
+% We provide some kernel-internal helpers for this. They will only be available if
+% \texttt{luatexbase} is available. This is not a big restriction though, because
+% ConTeXt (which does not use \texttt{luatexbase}) does not load expl3 in the format.
+%
+%    \begin{macrocode}
+local register_luadata, get_luadata
+
+if luatexbase then
+  local register = token.create'expl at luadata@bytecode'.index
+  if status.ini_version then
+%    \end{macrocode}
+%
+% \begin{macro}{register_luadata}
+% \texttt{register_luadata} is only available during format generation.
+% It accept a string which uniquely identifies the data object and has to be
+% provided to retrieve it later. Additionally it accepts a function which is
+% called in the \texttt{pre_dump} callback and which has to return a string that
+% evaluates to a valid Lua object to be preserved.
+%    \begin{macrocode}
+    local luadata, luadata_order = {}, {}
+
+    function register_luadata(name, func)
+      if luadata[name] then
+        error(string.format("LaTeX3 error: data name %q already in use", name))
+      end
+      luadata[name] = func
+      luadata_order[#luadata_order + 1] = func and name
+    end
+%    \end{macrocode}
+% \end{macro}
+% 
+% The actual work is done in \texttt{pre_dump}. The \texttt{luadata_order} is used
+% to ensure that the order is consistent over multiple runs.
+%    \begin{macrocode}
+luatexbase.add_to_callback("pre_dump", function()
+      if next(luadata) then
+        local str = "return {"
+        for i=1, #luadata_order do
+          local name = luadata_order[i]
+          str = string.format('%s[%q]=%s,', str, name, luadata[name]())
+        end
+        lua.bytecode[register] = assert(load(str .. "}"))
+      end
+    end, "latex3.luadata")
+  else
+%    \end{macrocode}
+%
+% \begin{macro}{get_luadata}
+% \texttt{get_luadata} is only available if data should be restored.
+% It accept the identifier which was used when the data object was registered and
+% returns the associated object. Every object can only be retrieved once.
+%    \begin{macrocode}
+    local luadata = lua.bytecode[register]
+    if luadata then
+      lua.bytecode[register] = nil
+      luadata = luadata()
+    end
+    function get_luadata(name)
+      if not luadata then return end
+      local data = luadata[name]
+      luadata[name] = nil
+      return data
+    end
+  end
+end
+%    \end{macrocode}
+% \end{macro}
+%
 %    \begin{macrocode}
 %</lua>
 %    \end{macrocode}
diff --git a/l3kernel/testfiles/m3intarray001.luatex.tlg b/l3kernel/testfiles/m3intarray001.luatex.tlg
index c05361920..ecbc49d9d 100644
--- a/l3kernel/testfiles/m3intarray001.luatex.tlg
+++ b/l3kernel/testfiles/m3intarray001.luatex.tlg
@@ -22,7 +22,7 @@ This is a coding error.
 LaTeX has been asked to create a new control sequence '\g_testa_intarray' but
 this name has already been used elsewhere.
 The current meaning is:
-  select font cmr10 at 0.00014pt
+  \long macro:->\s__intarray 9 
 Defining \g_testa_intarray on line ...
 ! LaTeX3 Error: Access to an entry beyond an array's bounds.
 For immediate help type H <return>.
@@ -30,18 +30,11 @@ For immediate help type H <return>.
 l. ...  }
 An attempt was made to access or store data at position 0 of the array
 '\g_testa_intarray', but this array has entries at positions from 1 to 12.
-! LaTeX3 Error: Integers larger than 2^{30}-1 cannot be stored in arrays.
-For immediate help type H <return>.
- ...                                              
-l. ...  }
-An attempt was made to store -2000000000 at position 1 in the array
-'\g_testa_intarray'. The largest allowed value -1073741823 will be used
-instead.
 ! LaTeX3 Error: Access to an entry beyond an array's bounds.
 For immediate help type H <return>.
  ...                                              
 l. ...  }
-An attempt was made to access or store data at position 13 of the array
+An attempt was made to access or store data at position 0 of the array
 '\g_testa_intarray', but this array has entries at positions from 1 to 12.
 ! Use of \??? doesn't match its definition.
 <argument> \???  
@@ -52,7 +45,7 @@ put `1' after `\a', since control sequence names are
 made up of letters only. The macro here has not been
 followed by the required stuff, so I'm ignoring it.
 0
--1073741823
+-2000000000
 0
 ! Use of \??? doesn't match its definition.
 <argument> \???  
@@ -68,41 +61,9 @@ followed by the required stuff, so I'm ignoring it.
 TEST 3: Unsafe array operations with errors
 ============================================================
 Defining \g_testb_intarray on line ...
-! Font \g_testb_intarray has only 15 fontdimen parameters.
-<recently read> \g_testb_intarray 
-l. ...  }
-To increase the number of font parameters, you must
-use \fontdimen immediately after the \font is loaded.
-! Dimension too large.
-<recently read> \c__intarray_sp_dim 
-l. ...  }
-I can't work with sizes bigger than about 19 feet.
-Continue and I'll use the largest value I can.
-! Dimension too large.
-<recently read> \c__intarray_sp_dim 
-l. ...  }
-I can't work with sizes bigger than about 19 feet.
-Continue and I'll use the largest value I can.
-! Font \g_testb_intarray has only 16 fontdimen parameters.
-<recently read> \g_testb_intarray 
-l. ...  }
-To increase the number of font parameters, you must
-use \fontdimen immediately after the \font is loaded.
-0
--1073741823
+-2000000000
 0
-123456
-0
-1234567
 Defining \g_testc_intarray on line ...
-! LaTeX3 Error: Size of array may not be negative: -1
-Type <return> to continue.
- ...                                              
-l. ...  }
-LaTeX does not know anything more about this error, sorry.
-Try typing <return> to proceed.
-If that doesn't work, type X <return> to quit.
-12345678
 ============================================================
 ============================================================
 TEST 4: Any stray non-zero?
@@ -127,19 +88,5 @@ l. ...  }
 TEST 6: Const
 ============================================================
 Defining \c_teste_intarray on line ...
-! LaTeX3 Error: Integers larger than 2^{30}-1 cannot be stored in arrays.
-For immediate help type H <return>.
- ...                                              
-l. ...  }
-An attempt was made to store -1073741824 at position \l__intarray_loop_int  in
-the array '\c_teste_intarray'. The largest allowed value -1073741823 will be
-used instead.
-! LaTeX3 Error: Integers larger than 2^{30}-1 cannot be stored in arrays.
-For immediate help type H <return>.
- ...                                              
-l. ...  }
-An attempt was made to store 1073741824 at position \l__intarray_loop_int  in
-the array '\c_teste_intarray'. The largest allowed value 1073741823 will be
-used instead.
-7,43,-1073741823,1073741823
+7,43,-1073741824,1073741824
 ============================================================
diff --git a/l3kernel/testfiles/m3intarray001.lvt b/l3kernel/testfiles/m3intarray001.lvt
index 298d9d87a..84e429061 100644
--- a/l3kernel/testfiles/m3intarray001.lvt
+++ b/l3kernel/testfiles/m3intarray001.lvt
@@ -69,7 +69,9 @@
   }
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\int_gadd:Nn \g__intarray_font_int { 100000 } % to make sure nothing is suppressed accidentally by scaling the font.
+\cs_if_exist:NT \g__intarray_font_int {
+  \int_gadd:Nn \g__intarray_font_int { 100000 } % to make sure nothing is suppressed accidentally by scaling the font.
+}
 \TEST { Any~stray~non-zero? }
   {
     \intarray_new:Nn \g_testd_intarray { 10 }
diff --git a/l3kernel/testfiles/m3skip001.tlg b/l3kernel/testfiles/m3skip001.luatex.tlg
similarity index 96%
copy from l3kernel/testfiles/m3skip001.tlg
copy to l3kernel/testfiles/m3skip001.luatex.tlg
index 6b00ba452..3ca538a9f 100644
--- a/l3kernel/testfiles/m3skip001.tlg
+++ b/l3kernel/testfiles/m3skip001.luatex.tlg
@@ -39,7 +39,7 @@ This is a coding error.
 LaTeX has been asked to create a new control sequence '\g_testa_dim' but this
 name has already been used elsewhere.
 The current meaning is:
-  \dimen168
+  \dimen167
 Defining \g_testa_dim on line ...
 \g_testa_dim=\dimen...
 ! LaTeX3 Error: Control sequence \g_testa_muskip already defined.
@@ -109,19 +109,19 @@ TEST 9: (SKIP) Overflow (BUT \maxdimen != maximum skip?!)
 -32767.00003pt plus -32767.00003pt minus -32767.00003pt (overflow by 1pt)
 ! Dimension too large.
 <to be read again> 
-                   p
+p
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
 ! Dimension too large.
 <to be read again> 
-                   m
+m
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
 ! Dimension too large.
 <to be read again> 
-                   \scan_stop: 
+\scan_stop: 
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
@@ -136,19 +136,19 @@ TEST 10: (SKIP) Underflow
 32767.00003pt plus 32767.00003pt minus 32767.00003pt (overflow by 1pt)
 ! Dimension too large.
 <to be read again> 
-                   p
+p
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
 ! Dimension too large.
 <to be read again> 
-                   m
+m
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
 ! Dimension too large.
 <to be read again> 
-                   \scan_stop: 
+\scan_stop: 
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
@@ -163,7 +163,7 @@ TEST 11: (DIM) Overflow (BUT \maxdimen != maximum dim?!)
 -32767.00003pt (overflow by 1pt)
 ! Dimension too large.
 <to be read again> 
-                   \__dim_eval_end: 
+\__dim_eval_end: 
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
@@ -178,7 +178,7 @@ TEST 12: (DIM) Underflow
 32767.00003pt (overflow by 1pt)
 ! Dimension too large.
 <to be read again> 
-                   \__dim_eval_end: 
+\__dim_eval_end: 
 l. ...}
 I can't work with sizes bigger than about 19 feet.
 Continue and I'll use the largest value I can.
diff --git a/l3kernel/testfiles/m3skip005.tlg b/l3kernel/testfiles/m3skip005.luatex.tlg
similarity index 89%
copy from l3kernel/testfiles/m3skip005.tlg
copy to l3kernel/testfiles/m3skip005.luatex.tlg
index eb73a74c4..52c4200e4 100644
--- a/l3kernel/testfiles/m3skip005.tlg
+++ b/l3kernel/testfiles/m3skip005.luatex.tlg
@@ -25,7 +25,7 @@ This is a coding error.
 LaTeX has been asked to create a new control sequence '\c_my_dim' but this
 name has already been used elsewhere.
 The current meaning is:
-  \dimen168
+  \dimen167
 Defining \c_my_dim on line ...
 \c_my_dim=\dimen...
 ! LaTeX3 Error: Control sequence \c_my_skip already defined.
@@ -59,7 +59,7 @@ TEST 3: Errors
 ============================================================
 ! Use of \??? doesn't match its definition.
 <argument> \???  
-                 ! LaTeX3 Error: '1.0pt2pt\scan_stop: ' in \dim_eval:n 
+      ! LaTeX3 Error: '1.0pt2pt\scan_stop: ' in \dim_eval:n 
 l. ...  }
 If you say, e.g., `\def\a1{...}', then you must always
 put `1' after `\a', since control sequence names are
@@ -68,7 +68,7 @@ followed by the required stuff, so I'm ignoring it.
 > 1pt2pt=1.0pt2pt\scan_stop: \__dim_eval_end: .
 ! Use of \??? doesn't match its definition.
 <argument> \???  
-                 ! LaTeX3 Error: '0.0ptpt)\scan_stop: ' in \dim_set:Nn 
+      ! LaTeX3 Error: '0.0ptpt)\scan_stop: ' in \dim_set:Nn 
 l. ...  }
 If you say, e.g., `\def\a1{...}', then you must always
 put `1' after `\a', since control sequence names are
@@ -76,7 +76,7 @@ made up of letters only. The macro here has not been
 followed by the required stuff, so I'm ignoring it.
 ! Use of \??? doesn't match its definition.
 <argument> \???  
-                 ! LaTeX3 Error: '0.769ptpt\scan_stop: ' in \dim_compare:nNnTF 
+      ! LaTeX3 Error: '0.769ptpt\scan_stop: ' in \dim_compare:nNnTF 
 l. ...  }
 If you say, e.g., `\def\a1{...}', then you must always
 put `1' after `\a', since control sequence names are
@@ -84,19 +84,19 @@ made up of letters only. The macro here has not been
 followed by the required stuff, so I'm ignoring it.
 ! Missing = inserted for \ifdim.
 <to be read again> 
-                   p
+p
 l. ...  }
 I was expecting to see `<', `=', or `>'. Didn't.
 ! Missing number, treated as zero.
 <to be read again> 
-                   p
+p
 l. ...  }
 A number should have been here; I inserted `0'.
 (If you can't figure out why I needed to see a number,
 look up `weird error' in the index to The TeXbook.)
 ! Illegal unit of measure (pt inserted).
 <to be read again> 
-                   m
+m
 l. ...  }
 Dimensions can be in units of em, ex, in, pt, pc,
 cm, mm, dd, cc, nd, nc, bp, or sp; but yours is a new one!
@@ -106,7 +106,7 @@ delete the erroneous units; e.g., type `2' to delete
 two letters. (See Chapter 27 of The TeXbook.)
 ! Use of \??? doesn't match its definition.
 <argument> \???  
-                 ! LaTeX3 Error: '1.0ptmu\scan_stop: ' in \skip_eval:n 
+      ! LaTeX3 Error: '1.0ptmu\scan_stop: ' in \skip_eval:n 
 l. ...  }
 If you say, e.g., `\def\a1{...}', then you must always
 put `1' after `\a', since control sequence names are
@@ -115,7 +115,7 @@ followed by the required stuff, so I'm ignoring it.
 > 1mu=1.0ptmu\scan_stop: \scan_stop: .
 ! Use of \??? doesn't match its definition.
 <argument> \???  
-                 ! LaTeX3 Error: '144.54ptches\scan_stop: ' in \skip_add:Nn 
+      ! LaTeX3 Error: '144.54ptches\scan_stop: ' in \skip_add:Nn 
 l. ...  }
 If you say, e.g., `\def\a1{...}', then you must always
 put `1' after `\a', since control sequence names are
@@ -123,7 +123,7 @@ made up of letters only. The macro here has not been
 followed by the required stuff, so I'm ignoring it.
 ! Use of \??? doesn't match its definition.
 <argument> \???  
-                 ! LaTeX3 Error: '1.0mu2\scan_stop: ' in \muskip_eval:n 
+      ! LaTeX3 Error: '1.0mu2\scan_stop: ' in \muskip_eval:n 
 l. ...  }
 If you say, e.g., `\def\a1{...}', then you must always
 put `1' after `\a', since control sequence names are
@@ -132,7 +132,7 @@ followed by the required stuff, so I'm ignoring it.
 > 1mu2=1.0mu2\scan_stop: \scan_stop: .
 ! Use of \??? doesn't match its definition.
 <argument> \???  
-                 ! LaTeX3 Error: '2.0mu)\scan_stop: ' in \muskip_add:Nn 
+      ! LaTeX3 Error: '2.0mu)\scan_stop: ' in \muskip_add:Nn 
 l. ...  }
 If you say, e.g., `\def\a1{...}', then you must always
 put `1' after `\a', since control sequence names are





More information about the latex3-commits mailing list.