[luatex] Memory leak in token.set_macro() implementation

[luigi scarso]

On Tue, 29 Oct 2024 at 07:49, user202729--- via luatex <luatex at tug.org>
wrote:

> I notice that repeated calls to `token.set_macro` consumes up memory, even
> though the macro themselves are freed.
>
> For example: run the following command
>
> ```
> lualatex '\directlua{for i=1,50000000 do token.set_macro("a", "")
> end}\stop' </dev/null
> ```
>
> This will consume a few GB of memory before terminating.
>
> After looking at the source code, the reason is because the reference
> count is not correctly zeroed, which leads to the token list not being
> freed when the macro is overwritten.
>
> The following patch should fix the issue.
>
>
> ```
> diff --git a/source/texk/web2c/luatexdir/lua/lnewtokenlib.c
> b/source/texk/web2c/luatexdir/lua/lnewtokenlib.c
> index 7c81e46..1b163b7 100644
> --- a/source/texk/web2c/luatexdir/lua/lnewtokenlib.c
> +++ b/source/texk/web2c/luatexdir/lua/lnewtokenlib.c
> @@ -1265,9 +1265,9 @@ static int set_macro(lua_State * L)
>          const char *se = str + lstr;
>          p = temp_token_head;
>          set_token_link(p, null);
> -        /* this left brace is used to store the number of arguments */
> -        fast_store_new_token(left_brace_token);
> -        /* and this ends the not present arguments, and no: we will not
> support arguments here*/
> +        /* reference count */
> +        fast_store_new_token(0);
> +        /* this ends the not present arguments, and no: we will not
> support arguments here*/
>          fast_store_new_token(end_match_token);
>          while (str < se) {
>              /* hh: str2uni could return len too (also elsewhere) */
> @@ -1334,7 +1334,7 @@ static int set_macro(lua_State * L)
>          halfword q; /* new node being added to the token list via
> |store_new_token| */
>          p = temp_token_head;
>          set_token_info(p,null);
> -        fast_store_new_token(left_brace_token);
> +        fast_store_new_token(0);
>          fast_store_new_token(end_match_token);
>          define(cs, call_cmd + (a % 4), token_link(temp_token_head));
>      }
> ```
>
> Please take a look. Thank you.
>
>
commit cd4888ec08e37cf0beae04be0a904d1df17eac41
Author: Luigi Scarso <luigi.scarso at gmail.com>
Date:   Sun Nov 3 13:17:40 2024 +0100

    Fixed a memory leak in token.put_next() and token.set_macro() (thanks
to user202729 at protonmail.com)

(also in texlive)

--
luigi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tug.org/pipermail/luatex/attachments/20241103/2557482f/attachment.htm>