[latex3-commits] [git/LaTeX3-latex3-latex3] main: Improve o-expansion of the output of \peek_analysis_map_inline:n (074572f6d)

[Bruno Le Floch]

Repository : https://github.com/latex3/latex3
On branch  : main
Link       : https://github.com/latex3/latex3/commit/074572f6d4bdf26e2e571f61621531be140ed2e9

>---------------------------------------------------------------

commit 074572f6d4bdf26e2e571f61621531be140ed2e9
Author: Bruno Le Floch <blflatex at gmail.com>
Date:   Mon Oct 3 22:47:53 2022 +0200

    Improve o-expansion of the output of \peek_analysis_map_inline:n
    
    Previously, we produced \exp_not:N \token for an expandable \token,
    which meant that o-expanding that would give a version of \token
    temporarily set to \relax.  Thus, \exp_after:wN \if_meaning:w #1
    would not match as expected.  Now we do
    \__kernel_exp_not:w \exp_after:wN { \exp_not:N #1 }
    to avoid that issue yet still deal with the possibility of outer
    macros.


>---------------------------------------------------------------

074572f6d4bdf26e2e571f61621531be140ed2e9
 l3kernel/CHANGELOG.md      |  3 +--
 l3kernel/l3tl-analysis.dtx | 25 +++++++++++++++++++++----
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/l3kernel/CHANGELOG.md b/l3kernel/CHANGELOG.md
index 2db59e1bb..a5b1842d9 100644
--- a/l3kernel/CHANGELOG.md
+++ b/l3kernel/CHANGELOG.md
@@ -8,8 +8,7 @@ this project uses date-based 'snapshot' version identifiers.
 ## [Unreleased]
 
 ### Changed
-- In `\peek_analysis_map_inline:n`, omit unnecessary `\exp_not:n` when the token
-  seen is a character that is neither active nor a macro parameter character
+- Usage of `\exp_not:n`/`\exp_not:N` in `\peek_analysis_map_inline:n` output
 
 ### Fixed
 - `\peek_analysis_map_inline:n` support for macro parameter characters (issue
diff --git a/l3kernel/l3tl-analysis.dtx b/l3kernel/l3tl-analysis.dtx
index cb5df54c4..eaf7459c3 100644
--- a/l3kernel/l3tl-analysis.dtx
+++ b/l3kernel/l3tl-analysis.dtx
@@ -169,6 +169,13 @@
 %     \cs{exp_not:n} \Arg{character} \cs{s_@@} \meta{hex catcode}
 %     \meta{char code} \cs{s_@@}.
 % \end{itemize}
+% In contrast, for \cs{peek_analysis_map_inline:n} we must allow for an
+% input stream containing \tn{outer} macros, so that wrapping all
+% control sequences in \cs{exp_not:n} is unsafe.  Instead, we write the
+% more elaborate \cs{__kernel_exp_not:w} \cs{exp_after:wN} |{|
+% \cs{exp_not:N} |\cs| |}|.  (On the other hand we make a better effort
+% by avoiding \cs{exp_not:n} for characters other than active and macro
+% parameters.)
 %
 % ^^A todo: ask LuaTeX list for an \ifx\undefined <active char>
 % ^^A which does not add the <active char> in memory.
@@ -1263,10 +1270,17 @@
 %   \cs{tl_set:Nx} to avoid grabbing it as an argument as |#1| may be
 %   \tn{outer}.  To allow~|#1| as an argument of the user's function
 %   (stored in \cs{l_@@_peek_code_tl}), we set it equal to
-%   \cs{scan_stop:} first, immediately before running the code as |#1|
+%   \cs{scan_stop:}, but we do it at the last minute because |#1|
 %   may be some pretty important function such as \cs{exp_after:wN}.
-%   Then we put the user's function and the first argument
-%   \cs{exp_not:N} |#1|.  Then we must add |{-1}0| if the token is a
+%   Then we put the user's function and the elaborate first argument
+%   \cs{__kernel_exp_not:w} \cs{exp_after:wN} |{| \cs{exp_not:N} |#1| |}|:
+%   indeed we cannot use \cs{exp_not:n} |{#1}| as this breaks for an
+%   \tn{outer} macro and we cannot use \cs{exp_not:N} |#1|, as
+%   \texttt{o}-expanding this yields a \enquote{notexpanded} token equal
+%   to (a weird) \tn{relax}, which would have the wrong value for
+%   primitive \TeX{} conditionals such as \cs{if_meaning:w}.
+%
+%   Then we must add |{-1}0| if the token is a
 %   control sequence and \Arg{charcode}|D| otherwise.  Distinguishing
 %   the two cases is easy: since we have made the escape character
 %   printable, \cs{token_to_str:N} gives at least two characters for a
@@ -1283,7 +1297,10 @@
       {
         \tex_let:D \exp_not:N #1 \scan_stop:
         \exp_not:o \l_@@_peek_code_tl
-        { \exp_not:N \exp_not:N \exp_not:N #1 }
+        {
+          \exp_not:n { \__kernel_exp_not:w \exp_after:wN }
+            { \exp_not:N \exp_not:N \exp_not:N #1 }
+        }
         \if:w \scan_stop:
               \exp_after:wN \use_none:n \token_to_str:N #1 \prg_do_nothing:
               \scan_stop: