[luatex] Adding a callback before trailing spaces are removed from a line of input

Sun Aug 29 21:42:00 CEST 2021

Dear Eduardo,

ah, light begins to shine! I suppose you could register a
process_line_buffer callback, which would `return ""` until it has
matched "End of block" or until it has been called five times, at which
point it would unregister itself.

Either way, this seems to be a solution. It's not a clean solution in
the sense that we have to read the TeX document twice, assuming that
it has not been altered in between, but it is a solution that we can
use today.

Thank you!
Vítek

On Sun, Aug 29, 2021 at 04:01:17PM -0300, Eduardo Ochs wrote:
> Hi Vítek,
> 
> the first prototype is here:
> 
>   https://github.com/edrx/markdown-dednat6/
> 
> The main part of test.tex is the one that contains:
> 
>   \directlua{Foo("End of block")}
>   Plop
>    Plip
> 
>     Bletch
>   End of block
> 
> The \directlua{Foo("End of block")} prints the lines before "End of
> block" to stdout with line numbers, and you should get this on stdout:
> 
>   39:     "Plop"
>   40:     " Plip"
>   41:     "   "
>   42:     "  Bletch"
> 
> but I don't remember how to make luatex skip the next 5 lines of
> input.. how can we do that? I need that for the second prototype, to
> convince you that we can use that to parse Markdown... =)
> 
>   Cheers,
>     Eduardo Ochs
>     http://angg.twu.net/dednat6.html
> 
> 
> On Sun, 29 Aug 2021 at 12:57, Vítek Novotný <witiko at mail.muni.cz> wrote:
> 
> > On Sun, Aug 29, 2021 at 08:03:47AM -0300, Eduardo Ochs wrote:
> > > Hi Vitek,
> > >
> > > I am the author of this package,
> > >
> > >   http://angg.twu.net/dednat6/tug-slides.pdf
> > >   http://angg.twu.net/dednat6/tugboat-rev2.pdf
> > >   http://angg.twu.net/dednat6.html
> >
> > Dear Eduardo,
> >
> > thank you for your response. What a beautiful package: ASCII art
> > diagrams translated to production-quality outputs?! Sold! Perhaps we
> > should write an article about using markdown and friends to produce
> > academic texts using easy-to-read-and-write domain-specific languages.
> >
> > > that uses Lua to read material from the _comments_ of the current .tex
> > > file. I think that what you need can be implemented easily on top of
> > > this single-file version of the core of dednat6,
> > >
> > >   http://angg.twu.net/dednat6/dednat6/minimalcore.lua.html
> > >   http://angg.twu.net/dednat6/dednat6/minimalcore.lua
> > >   http://angg.twu.net/dednat6/demo-core.tex.html
> > >   http://angg.twu.net/dednat6/demo-core.tex
> > >   http://angg.twu.net/dednat6/demo-core.pdf
> > >
> > > that only implements the code that interprets each block of lines
> > > starting with "%L" as a chunk of Lua code...
> > >
> > > If this looks interesting to you, please get in touch! I took a
> > > quick look at your markdown.dtx and I got the impression that 1) it
> > > would be trivial to make my minimalcore.lua read the markdown code in
> > > fencedCode blocks and make TeX skip over it, and that 2) it would be
> > > easy to port the ideas from the prototype made with minimalcore.lua to
> > > your markdown package.
> >
> > Using an external program to preprocess a TeX document is an interesting
> > approach, but one that doesn't seem compatible with the way the Markdown
> > package has operated for the past five years: we buffer input inside an
> > environment, and we pass it to the Lua parser. We definitely don't want
> > to use comments to separate Markdown text from the rest of the document:
> > That can be justified for short ASCII art diagrams, but sounds pretty
> > user-hostile for, say, a book worth of markdown text.
> >
> > >   Cheers,
> > >     Eduardo Ochs
> > >     http://angg.twu.net/contact.html
> >
> > Best,
> > Vítek
> >
> > > On Sat, 28 Aug 2021 at 19:57, Vítek Novotný <witiko at mail.muni.cz> wrote:
> > >
> > > > Hello all,
> > > >
> > > > in Knuth's TeX, trailing spaces are removed very early on when a line
> > is
> > > > being put to the input buffer. [1]  According to Eijkhout's TeX by
> > > > Topic, this is because "these spaces are hard to see in an editor" [2].
> > > >
> > > >  [1]: https://texdoc.org/serve/tex.pdf/0#page=15
> > > >  [2]: http://mirrors.ctan.org/info/texbytopic/TeXbyTopic.pdf
> > > >
> > > > I develop and maintain the Markdown package [3] for plain TeX, ConTeXt,
> > > > and LaTeX. The package makes it possible to use the lightweight markup
> > > > of markdown [4] in TeX documents. In markdown, a hard line break can be
> > > > inserted by ending a line with two or more spaces. However, since
> > > > trailing spaces are removed by TeX, hard breaks are only recognized
> > when
> > > > we' are inserting an external markdown file, not when markdown is typed
> > > > in the top-level document. This deficiency is known and documented [5],
> > > > but I am hoping we could resolve it with LuaTeX.
> > > >
> > > >  [3]: https://github.com/witiko/markdown
> > > >  [4]: https://daringfireball.net/projects/markdown
> > > >  [5]:
> > > > https://mirrors.ctan.org/macros/generic/markdown/markdown.pdf#page=20
> > > >
> > > > In LuaTeX, the `process_input_buffer` callback [6] can be used to
> > > > intercept the text coming *out* of the input buffer. However, the
> > > > trailing spaces have already been removed by this point.
> > > >
> > > > By adding a callback right after a line has entered the input buffer
> > > > [1], we could either replace the trailing space characters with tabs,
> > > > or place a character such as the zero-width non-joiner (U+200C) to the
> > > > right of the trailing spaces.
> > > >
> > > >  [6]: https://www.pragma-ade.com/general/manuals/luatex.pdf#page=176
> > > >
> > > > Is this something you would consider---if not for LuaTeX then perhaps
> > > > for LuaMetaTeX?
> > > >
> > > > Best regards,
> > > > Vítek Novotný
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://tug.org/pipermail/luatex/attachments/20210829/badc7f21/attachment.sig>