[luatex] Adding a callback before trailing spaces are removed from a line of input

[Vítek Novotný]

Hello all,

in Knuth's TeX, trailing spaces are removed very early on when a line is
being put to the input buffer. [1]  According to Eijkhout's TeX by
Topic, this is because "these spaces are hard to see in an editor" [2].

 [1]: https://texdoc.org/serve/tex.pdf/0#page=15
 [2]: http://mirrors.ctan.org/info/texbytopic/TeXbyTopic.pdf

I develop and maintain the Markdown package [3] for plain TeX, ConTeXt,
and LaTeX. The package makes it possible to use the lightweight markup
of markdown [4] in TeX documents. In markdown, a hard line break can be
inserted by ending a line with two or more spaces. However, since
trailing spaces are removed by TeX, hard breaks are only recognized when
we' are inserting an external markdown file, not when markdown is typed
in the top-level document. This deficiency is known and documented [5],
but I am hoping we could resolve it with LuaTeX.

 [3]: https://github.com/witiko/markdown
 [4]: https://daringfireball.net/projects/markdown
 [5]: https://mirrors.ctan.org/macros/generic/markdown/markdown.pdf#page=20

In LuaTeX, the `process_input_buffer` callback [6] can be used to
intercept the text coming *out* of the input buffer. However, the
trailing spaces have already been removed by this point.

By adding a callback right after a line has entered the input buffer
[1], we could either replace the trailing space characters with tabs,
or place a character such as the zero-width non-joiner (U+200C) to the
right of the trailing spaces.

 [6]: https://www.pragma-ade.com/general/manuals/luatex.pdf#page=176

Is this something you would consider---if not for LuaTeX then perhaps
for LuaMetaTeX?

Best regards,
Vítek Novotný
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://tug.org/pipermail/luatex/attachments/20210829/1c1f79bc/attachment.sig>