[luatex] Using 'append_to_vlist_filter' callback causes stack overflow on Lua side

[Hironori KITAGAWA]

Hello all,

I'm looking into 
https://tex.stackexchange.com/questions/652379/warning-hpack-filter-error-xmf-dist-tex-luatex-luaotfload-fontloader-20-2,

Iit seems that using the 'append_to_vlist_filter' callback with a large
number of paragraphs causes a stack overflow.

---- source file (t45298-2.tex)
% plain LuaTeX or LuaLaTeX
\ifdefined\documentclass
  \documentclass{minimal}\begin{document}
\expandafter\else\input luatexbase.sty
\expandafter\fi
\directlua{
  luatexbase.add_to_callback('append_to_vlist_filter',
    function (b, loc, prev, mirrored) node.write(b); return nil end,
  'test', 1)}

\newcount\foo
\def\A{a\space\par
  \advance\foo1 \ifnum\foo<500000 \expandafter\A\fi}
\A
\end{document}
----

---- plain LuaTeX
This is LuaTeX, Version 1.15.1 (TeX Live 2023/dev) 
 restricted system commands enabled.
(./t45298-2.tex
(/opt/texlive/2022/texmf-dist/tex/luatex/luatexbase/luatexbase.sty
(/opt/texlive/2022/texmf-dist/tex/luatex/ctablestack/ctablestack.sty
(/opt/texlive/2022/texmf-dist/tex/latex/base/ltluatex.tex))) [1{/opt/texlive/20
22/texmf-var/fonts/map/pdftex/updmap/pdftex.map}] [2]
... [2192]
warning  (append to vlist): error: [\directlua]:1: stack overflow
.
\A ->a\space \par 
                  \advance \foo 1 \ifnum \foo <500000 \expandafter \A
\fi l.13 \A
     
?
----
Processing t45298-2.tex with lualatex also causes the overflow:
---- LuaLaTeX
... [2583]
warning  (node filter): error: ...xmf-dist/tex/luatex/luaotfload/fontloader-202
2-06-15.lua:25611: stack overflow

.
<argument> ...ype:D \tex_hskip:D \c_zero_dim \fi: \tex_par:D 
                                                  \hook_use:n {para/after}\@...

l.13 \A
     
?
----

I found that there isn't "lua_settop(Luas, s_top);" before "return 1;" 
statement of the function lua_appendtovlist_callback() (lua/luanodes.c).
Is this related to the overflow?

Regards 
Hironori KITAGAWA
<h_kitagawa2001 at yahoo.co.jp>