[luatex] Adding a callback before trailing spaces are removed from a line of input

[Eduardo Ochs]

Hi Vítek,

the first prototype is here:

  https://github.com/edrx/markdown-dednat6/

The main part of test.tex is the one that contains:

  \directlua{Foo("End of block")}
  Plop
   Plip

    Bletch
  End of block

The \directlua{Foo("End of block")} prints the lines before "End of
block" to stdout with line numbers, and you should get this on stdout:

  39:     "Plop"
  40:     " Plip"
  41:     "   "
  42:     "  Bletch"

but I don't remember how to make luatex skip the next 5 lines of
input.. how can we do that? I need that for the second prototype, to
convince you that we can use that to parse Markdown... =)

  Cheers,
    Eduardo Ochs
    http://angg.twu.net/dednat6.html


On Sun, 29 Aug 2021 at 12:57, Vítek Novotný <witiko at mail.muni.cz> wrote:

> On Sun, Aug 29, 2021 at 08:03:47AM -0300, Eduardo Ochs wrote:
> > Hi Vitek,
> >
> > I am the author of this package,
> >
> >   http://angg.twu.net/dednat6/tug-slides.pdf
> >   http://angg.twu.net/dednat6/tugboat-rev2.pdf
> >   http://angg.twu.net/dednat6.html
>
> Dear Eduardo,
>
> thank you for your response. What a beautiful package: ASCII art
> diagrams translated to production-quality outputs?! Sold! Perhaps we
> should write an article about using markdown and friends to produce
> academic texts using easy-to-read-and-write domain-specific languages.
>
> > that uses Lua to read material from the _comments_ of the current .tex
> > file. I think that what you need can be implemented easily on top of
> > this single-file version of the core of dednat6,
> >
> >   http://angg.twu.net/dednat6/dednat6/minimalcore.lua.html
> >   http://angg.twu.net/dednat6/dednat6/minimalcore.lua
> >   http://angg.twu.net/dednat6/demo-core.tex.html
> >   http://angg.twu.net/dednat6/demo-core.tex
> >   http://angg.twu.net/dednat6/demo-core.pdf
> >
> > that only implements the code that interprets each block of lines
> > starting with "%L" as a chunk of Lua code...
> >
> > If this looks interesting to you, please get in touch! I took a
> > quick look at your markdown.dtx and I got the impression that 1) it
> > would be trivial to make my minimalcore.lua read the markdown code in
> > fencedCode blocks and make TeX skip over it, and that 2) it would be
> > easy to port the ideas from the prototype made with minimalcore.lua to
> > your markdown package.
>
> Using an external program to preprocess a TeX document is an interesting
> approach, but one that doesn't seem compatible with the way the Markdown
> package has operated for the past five years: we buffer input inside an
> environment, and we pass it to the Lua parser. We definitely don't want
> to use comments to separate Markdown text from the rest of the document:
> That can be justified for short ASCII art diagrams, but sounds pretty
> user-hostile for, say, a book worth of markdown text.
>
> >   Cheers,
> >     Eduardo Ochs
> >     http://angg.twu.net/contact.html
>
> Best,
> Vítek
>
> > On Sat, 28 Aug 2021 at 19:57, Vítek Novotný <witiko at mail.muni.cz> wrote:
> >
> > > Hello all,
> > >
> > > in Knuth's TeX, trailing spaces are removed very early on when a line
> is
> > > being put to the input buffer. [1]  According to Eijkhout's TeX by
> > > Topic, this is because "these spaces are hard to see in an editor" [2].
> > >
> > >  [1]: https://texdoc.org/serve/tex.pdf/0#page=15
> > >  [2]: http://mirrors.ctan.org/info/texbytopic/TeXbyTopic.pdf
> > >
> > > I develop and maintain the Markdown package [3] for plain TeX, ConTeXt,
> > > and LaTeX. The package makes it possible to use the lightweight markup
> > > of markdown [4] in TeX documents. In markdown, a hard line break can be
> > > inserted by ending a line with two or more spaces. However, since
> > > trailing spaces are removed by TeX, hard breaks are only recognized
> when
> > > we' are inserting an external markdown file, not when markdown is typed
> > > in the top-level document. This deficiency is known and documented [5],
> > > but I am hoping we could resolve it with LuaTeX.
> > >
> > >  [3]: https://github.com/witiko/markdown
> > >  [4]: https://daringfireball.net/projects/markdown
> > >  [5]:
> > > https://mirrors.ctan.org/macros/generic/markdown/markdown.pdf#page=20
> > >
> > > In LuaTeX, the `process_input_buffer` callback [6] can be used to
> > > intercept the text coming *out* of the input buffer. However, the
> > > trailing spaces have already been removed by this point.
> > >
> > > By adding a callback right after a line has entered the input buffer
> > > [1], we could either replace the trailing space characters with tabs,
> > > or place a character such as the zero-width non-joiner (U+200C) to the
> > > right of the trailing spaces.
> > >
> > >  [6]: https://www.pragma-ade.com/general/manuals/luatex.pdf#page=176
> > >
> > > Is this something you would consider---if not for LuaTeX then perhaps
> > > for LuaMetaTeX?
> > >
> > > Best regards,
> > > Vítek Novotný
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://tug.org/pipermail/luatex/attachments/20210829/35ae42fb/attachment.html>