[luatex] Adding a callback before trailing spaces are removed from a line of input

[Vítek Novotný]

On Sun, Aug 29, 2021 at 08:03:47AM -0300, Eduardo Ochs wrote:
> Hi Vitek,
> 
> I am the author of this package,
> 
>   http://angg.twu.net/dednat6/tug-slides.pdf
>   http://angg.twu.net/dednat6/tugboat-rev2.pdf
>   http://angg.twu.net/dednat6.html

Dear Eduardo,

thank you for your response. What a beautiful package: ASCII art
diagrams translated to production-quality outputs?! Sold! Perhaps we
should write an article about using markdown and friends to produce
academic texts using easy-to-read-and-write domain-specific languages.

> that uses Lua to read material from the _comments_ of the current .tex
> file. I think that what you need can be implemented easily on top of
> this single-file version of the core of dednat6,
> 
>   http://angg.twu.net/dednat6/dednat6/minimalcore.lua.html
>   http://angg.twu.net/dednat6/dednat6/minimalcore.lua
>   http://angg.twu.net/dednat6/demo-core.tex.html
>   http://angg.twu.net/dednat6/demo-core.tex
>   http://angg.twu.net/dednat6/demo-core.pdf
> 
> that only implements the code that interprets each block of lines
> starting with "%L" as a chunk of Lua code...
> 
> If this looks interesting to you, please get in touch! I took a
> quick look at your markdown.dtx and I got the impression that 1) it
> would be trivial to make my minimalcore.lua read the markdown code in
> fencedCode blocks and make TeX skip over it, and that 2) it would be
> easy to port the ideas from the prototype made with minimalcore.lua to
> your markdown package.

Using an external program to preprocess a TeX document is an interesting
approach, but one that doesn't seem compatible with the way the Markdown
package has operated for the past five years: we buffer input inside an
environment, and we pass it to the Lua parser. We definitely don't want
to use comments to separate Markdown text from the rest of the document:
That can be justified for short ASCII art diagrams, but sounds pretty
user-hostile for, say, a book worth of markdown text.

>   Cheers,
>     Eduardo Ochs
>     http://angg.twu.net/contact.html

Best,
Vítek

> On Sat, 28 Aug 2021 at 19:57, Vítek Novotný <witiko at mail.muni.cz> wrote:
> 
> > Hello all,
> >
> > in Knuth's TeX, trailing spaces are removed very early on when a line is
> > being put to the input buffer. [1]  According to Eijkhout's TeX by
> > Topic, this is because "these spaces are hard to see in an editor" [2].
> >
> >  [1]: https://texdoc.org/serve/tex.pdf/0#page=15
> >  [2]: http://mirrors.ctan.org/info/texbytopic/TeXbyTopic.pdf
> >
> > I develop and maintain the Markdown package [3] for plain TeX, ConTeXt,
> > and LaTeX. The package makes it possible to use the lightweight markup
> > of markdown [4] in TeX documents. In markdown, a hard line break can be
> > inserted by ending a line with two or more spaces. However, since
> > trailing spaces are removed by TeX, hard breaks are only recognized when
> > we' are inserting an external markdown file, not when markdown is typed
> > in the top-level document. This deficiency is known and documented [5],
> > but I am hoping we could resolve it with LuaTeX.
> >
> >  [3]: https://github.com/witiko/markdown
> >  [4]: https://daringfireball.net/projects/markdown
> >  [5]:
> > https://mirrors.ctan.org/macros/generic/markdown/markdown.pdf#page=20
> >
> > In LuaTeX, the `process_input_buffer` callback [6] can be used to
> > intercept the text coming *out* of the input buffer. However, the
> > trailing spaces have already been removed by this point.
> >
> > By adding a callback right after a line has entered the input buffer
> > [1], we could either replace the trailing space characters with tabs,
> > or place a character such as the zero-width non-joiner (U+200C) to the
> > right of the trailing spaces.
> >
> >  [6]: https://www.pragma-ade.com/general/manuals/luatex.pdf#page=176
> >
> > Is this something you would consider---if not for LuaTeX then perhaps
> > for LuaMetaTeX?
> >
> > Best regards,
> > Vítek Novotný
> >
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://tug.org/pipermail/luatex/attachments/20210829/dfeeee6c/attachment-0001.sig>