texlive[52191] Master/texmf-dist: tokcycle (27sep19)
commits+karl at tug.org
commits+karl at tug.org
Fri Sep 27 23:24:43 CEST 2019
Revision: 52191
http://tug.org/svn/texlive?view=revision&revision=52191
Author: karl
Date: 2019-09-27 23:24:43 +0200 (Fri, 27 Sep 2019)
Log Message:
-----------
tokcycle (27sep19)
Modified Paths:
--------------
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
===================================================================
(Binary files differ)
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex 2019-09-27 21:24:30 UTC (rev 52190)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex 2019-09-27 21:24:43 UTC (rev 52191)
@@ -44,21 +44,20 @@
\newcytoks{<new token register>}
\end{myverbbox}
-{\centering{\LARGE Package \TokCycle{}\par}
+{\centering{\LARGE Package \TokCycle{} \Large (v\tcver)\par}
Steven B Segletes
{\footnotesize $<$steven.b.segletes.civ at mail.mil$>$}
-\\\small{contributor\footnote{
-I am extremely grateful to Christian
+\\\small{contributor: Christian Tellechea}\footnote{
+{I am extremely grateful to Christian $<$unbonpetit at netc.fr$>$}
for his assistance in the development of this package.
The \macname{addcytoks} macro was provided by him.
-He gave constant reminders on what the token parser should be able
+He gave constant reminders on what the parser should be able
to achieve, thus motivating me to spend the extra time striving for
- a generality of application that would not otherwise have made it
- into the package.
+ a generality of application that would not come naturally to me.
I value highly his collegiality and hold his expertise in the highest
regard.
}%
-: Christian Tellechea {\footnotesize$<$unbonpetit at netc.fr$>$}}\\\today\par}
+\\\today\par}
\begin{quote}
The \TokCycle{} package helps one to build tools to process
@@ -90,10 +89,12 @@
The purpose of the \TokCycle{} package is to provide a tool
to assist the user in developing token-processing macros and
environments.
-The types of processing is limited only by the creativity of the user,
+The types of processing are limited only by the creativity of the user,
but examples might include letter-case-operations, letter spacing,
dimensional manipulation,
simple-ciphering, \{group\} manipulation, macro removal, etc.
+In one sense, it can be thought of as a streaming editor that operates
+ on \LaTeX{} input streams.
The package can be loaded into both plain \TeX{}, by way of the
invocation \verb|\input tokcycle.tex|
@@ -134,8 +135,8 @@
\end{itemize}
-The basic approach of all macros/environments is to grab each
- successive token (or group) from the input stream,
+The basic approach of all \TokCycle{} macros/environments is to
+ grab each successive token (or group) from the input stream,
decide what category it is, and use the currently active
processing directives to make any desired transformation of the
token (or group).
@@ -151,6 +152,18 @@
The \TokCycle{} categories by which the input stream
is dissected include \Character{},
\Group{}, \Macro{}, and \Space{}.
+
+Catcode-0 tokens are directed to the \Macro{} directive for processing.
+Catcode-10 tokens are directed to the \Space{} directive.
+When an explicit catcode-1 token is encountered in the \TokCycle{} input
+ stream, the contents of the entire group (sans the grouping) are directed
+ to the \Group{} directive for further processing, which may in turn,
+ redirect the individual tokens to the other categories.
+The handling options of implicit cat-1 and 2 tokens are described later in
+ this document.
+Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted,
+ are directed to the \Character{} directive for processing.
+
The syntax of the non-\texttt{xpress} macros/environments is\medskip
\begin{verbbox}[\vbdelim]
@@ -195,7 +208,7 @@
{<Character processing directive>}
{<Group-content processing directive>}
{<Macro processing directive>}
- {<Space processing directive>}%
+ {<Space processing directive>}
\end{verbbox}
\medskip\theverbbox\medskip\par\noindent
This will then permit simplified invocations of the form
@@ -241,14 +254,17 @@
The interpretation of these directives will be explained in the
remainder of this document.
+Let it suffice for now to say that the default directive settings
+ pass through the input stream to the output stream, without
+ change.
\section{Commands in the \TokCycle{} directives}\label{s:coms}
-The command-line token cycling tools to be employed in the package were
+The command-line token cycling tools provided in the package are
listed in section~\ref{s:intro}.
-For each of those commands/pseudo-environments provided, the
+For each of those commands and/or pseudo-environments, the
user must (explicitly or implicitly) detail a set of directives to
specify the manner in which the \Character{}, \Group{}, \Macro{},
and \Space{} tokens found in the input stream are to be processed.
@@ -310,8 +326,9 @@
Grouping in the output stream is determined by the externally set
condition \macname{ifstripgrouping}.
-The package default is \macname{stripgroupingfalse}, such that any grouping
- that appears in the input stream will be echoed in the output stream.
+The package default is \macname{stripgroupingfalse}, such that any
+ explicit grouping that appears in the input stream will be echoed
+ in the output stream.
The alternative, \macname{stripgroupingtrue}, is dangerous
in the sense that it will strip the cat-1,2 grouping from the
group's tokens, thereby
@@ -356,7 +373,8 @@
\subsection{Escaping content from \TokCycle{} processing\label{s:escape}}
There are times you may wish to prevent tokens in the \TokCycle{}
- input stream from being operated on by \TokCycle; you just want
+ input stream from being operated on by \TokCycle.
+Rather, you just want
the content passed through unchanged to the output; that is,
with the intent to have multi-token content bypass the
\TokCycle{} directives altogether.
@@ -363,9 +381,9 @@
\begin{sloppypar}
The method developed by the package is to enclose the escaped
- content in the input stream with the \TokCycle{} escape character,
- initially set to a vertical rule character found on the keyboard:
- ``$|\mkern1mu$''.
+ content in the input stream between a set of \TokCycle{} escape
+ characters, initially set to a vertical rule character found
+ on the keyboard: ``$|\mkern1mu$''.
The main proviso is that the escaped content cannot straddle a
group boundary (the complete group may, however, be escaped).
The escape character can be changed with
@@ -383,30 +401,67 @@
directive (generally the \Character{} directive), to direct the
course of action within the directive.
-\subsubsection{Active characters: \macname{ifactivetok}}
+\subsubsection{Active characters}
-Active (cat-13) characters that occur in the input stream result in the
+\paragraph{\macname{ifactivetok}:}
+
+Active (cat-13) tokens that occur in the input stream result in the
flag \macname{ifactivetok} being set \macname{activetoktrue}.
-Note that the substitution of the token's active value occurs \textit{after}
- \TokCycle{} processing, and so \TokCycle{} only sees it as
- \macname{active}, regardless of its later substitution or assignment.
+Note that the expansion of the token's active \macname{def}
+ occurs \textit{after} \TokCycle{} processing.
+With active \macname{let}'s, there is no text substitution; however,
+ the assignment is already active at the time of \TokCycle{} processing.
+%
The only exception to this rule is with pre-expanded input,
\macname{expandedtokcycle[xpress]}.
-If an active token's substitution is governed by a \macname{def} (but
- not \macname{let}), the text substitution will have occurred before
- reaching the token cycle.
+If an active token's substitution is governed by a \macname{def}, the
+ text substitution will have occurred before reaching the token cycle.
+\paragraph{\macname{ifactivetokunexpandable}:}
+
+This flag is similar to \macname{ifactivetok}, in that a token must
+ be active for this to be set true.
+However, in addition, it is only
+ true if the active token is \macname{let} to a character or a
+ primitive, neither of which can be expanded.
+Active characters assigned via \macname{def} or else \macname{let} to a
+ macro will \textit{not} qualify as \macname{activetokunexpandabletrue}.
+
+\paragraph{\macname{ifactivechar}:}
+
+This flag, rather than testing the token, tests the character code of
+ the token, to see if it is set active.
+Generally, the token and its character code will be syncronized
+ in their \textit{activeness}.
+However, if a token is tokenized when active, but the corresponding character
+ code is made non-active in the meantime, prior to the token reaching
+ \TokCycle{} processing, this flag will be set \macname{activecharfalse}.
+A similar discrepency will arise if a token is not active when tokenized,
+ but the character code is made active in the interim, prior to
+ \TokCycle{} processing.
+
\subsubsection{Implicit tokens: \macname{ifimplicittok}}
-Implicit tokens, those assigned via \macname{let}, are flagged with
+Implicit tokens, those assigned via \macname{let} to characters\footnote{%
+Some clarification may be needed.
+Control sequences and active characters that are \macname{let}
+ to something other than a cat-0 control sequence will be flagged
+ as implicit.
+If implicit, a token will be processed
+ through the \Character{} directive (exceptions noted).
+On the other hand, if a control sequence or active character is \macname{let}
+ to a cat-0 control sequence, it will be directed to the \Macro{}
+ directive for processing, without the implicit flag.
+}%
+ , are flagged with
a \texttt{true} setting for \macname{ifimplicittok}.
Generally, implicit tokens will be directed to the \Character{}
directive for processing.
There are, however, two exceptions:
-i) implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
+i)~implicit grouping tokens (e.g., \macname{bgroup} and \macname{egroup})
will not appear in any directive unless the
\macname{stripimplicitgroupingcase} has been set to $-$1; and
-ii) implicit space tokens (e.g., \macname{@sptoken}) will be processed
+ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed
by the \Space{} directive.
\subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}}
@@ -425,13 +480,9 @@
taken), explicit cat-6 characters are converted
into the identical character with category code of 12.
On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|)
- are converted into a fixed-name implicit macro, \macname{implicitsixtok},
+ are converted into a fixed-name macro, \macname{implicitsixtok},
whose cat-12 substitution text is a \macname{string} of the original
implicit-macro name.
-
-As previously mentioned, active characters whose active value is
- \macname{let} to cat-6 (e.g., \verb|\catcode`X=\active \letX#|) will be
- flagged as \macname{activetoktrue}, but not \macname{catSIXtrue}.
\end{sloppypar}
\subsubsection{Parameters (\texttt{\#1}--\texttt{\#9}):
@@ -466,7 +517,7 @@
of proper expansion and execution.
These four restructured \textit{if} macros came in handy writing the
package, and may be of use in creating your directives,
- to prevent \macname{else} and \macname{fi} from getting in the way:\medskip
+ without \macname{else} and \macname{fi} getting in the way:\medskip
\begin{verbbox}[\vbdelim]
\tctestifcon{<TeX-if-condition>}{<true-code>}{<false-code>}
@@ -495,7 +546,8 @@
by \macname{aftertokcycle}, as they actually set it locally
(within their scope) to accomplish their own purposes.
-\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}}
+\subsection{Accommodating catcode-1,2 changes: \macname{settcGrouping}%
+ \label{s:grouping}}
In order to avoid making the \TokCycle{} parser overly complex, requiring
multiple passes of the input stream, the package defaults to using
@@ -518,12 +570,10 @@
invoke \verb|\settcGrouping<<#1>>|.
These will then be the grouping tokens in the \TokCycle{} output
stream until set to something else.
-The grouping tokens are \textit{not} reset as part of
- \macname{resettokcycle}.
\section{Usage Examples}
-See the adjunct file, \texttt{tokcycle-examples.pdf}, for a wide
+See the adjunct file, \texttt{tokcycle-examples.pdf}, for an
array of \TokCycle{} examples.
\section{Summary of known package limitations}
@@ -549,9 +599,11 @@
bracing group; however,
grouping will still be handled properly (i.e., cat-1,2 tokens will
be detected), even if the package is not notified.
+See section~\ref{s:grouping}.
\item Should one need to keep track of the \textit{names} of implicit tokens,
- then only one named implicit cat-6 token may appear in the input
+ then only one named implicit cat-6 token (e.g., \verb|\let\svhash#|)
+ may appear in the input
stream (though it can appear multiple times).
There is no limit on explicit cat-6 toks.
This limitation occurs because implicit cat-6 tokens are converted
@@ -581,11 +633,9 @@
\section*{Source Code}
\Large\texttt{tokcycle.sty}
+\verbfilenobox[\footnotesize]{tokcycle.sty}\vspace{2in}
-\verbfilenobox[\footnotesize]{tokcycle.sty}\bigskip
-
\noindent\texttt{tokcycle.tex}
-
\verbfilenobox[\footnotesize]{tokcycle.tex}
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
===================================================================
(Binary files differ)
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex 2019-09-27 21:24:30 UTC (rev 52190)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex 2019-09-27 21:24:43 UTC (rev 52191)
@@ -50,13 +50,13 @@
\def\altdecytoks{\par\medskip\begingroup\noindent\macname{cytoks}
\ \textit{alt}detokenization:\\\raggedright\footnotesize\ttfamily
\expandafter\altdetokenize\expandafter{\the\cytoks}\par\endgroup}
-\newcommand\TokCycle[1][t]{\textsf{#1okcycle}}
+\newcommand\TokCycle{\textsf{tokcycle}}
\parindent0pt
\parskip0pt
\begin{document}
{\centering\LARGE \TokCycle{} Package Examples\par\smallskip
-\large Steven B. Segletes\\\today\par}
+\large\today\par}
\tableofcontents
@@ -63,8 +63,99 @@
\makeatletter
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% ALREADY PLACED IN tokcycle.tex
+%
+%\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{\trapactivelet#1}}
+%
+%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can at absorb@@
+%\def\trapactivelet#1{\tctestifcon{%
+% \expandafter\if\detokenize{#1}#1}{}{\activetoktrue\implicittoktrue}}
+%
+%\makeatletter
+%\long\def\backslashcmds#1{%
+% \test at ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}%
+% {\implicittoktrue\trapimplicitegrp#1\tctestifcon{\iftc at implicitgrp}%
+% {\ifcase\@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}%
+% {\@chrT#1}}}
+%
+%\long\def\can at absorb@#1{\tctestifnum{\count at stringtoks{#1}>1}%
+% {\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}%
+% {\trapactivetok#1\tc at trapescape#1{\tc at escapecytoks}{\can at absorb@@#1}}}
+%
+%\long\def\can at absorb@@#1{\let\@tmp#1\test at ifmacro\@tmp
+% {\implicittokfalse\@macT#1}{\@chrT#1}\@tokcycle}
+%
+%\long\def\count at toks#1{\the\numexpr-1\count@@toks#1.\tc at endcnt}
+%\long\def\count@@toks#1#2\tc at endcnt{+1\tc at ifempty{#2}{\relax}{\count@@toks#2\tc at endcnt}}
+%
+% END ALREADY PLACED IN tokcycle.tex
+% REVISED ACTIVE TOK TRAP
+%\newif\ifactivechar
+%\newif\ifactivetokunexpandable
+%\def\tc at resetifs{\activetokfalse\implicittokfalse\tc at implicitgrpfalse
+% \catSIXfalse\activecharfalse\activetokunexpandablefalse}
+
+%\long\def\can at absorb@#1{\tctestifnum{\count at stringtoks{#1}>1}%
+% {\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}%
+% {\trapactives#1\tc at trapescape#1{\tc at escapecytoks}{\can at absorb@@#1}}}
+
+% \activechartrue: \catcode`X=\active AT TIME OF TEST
+% \activetoktrue: \catcode`X=\active AT TIME OF X's TOKENIZATION
+% \activetokunexpandabletrue: \catcode`X=\active WHEN TOKENIZED AND
+% \let XY or \let X\primitive AT TIME OF TEST
+% \implicittoktrue \let(X or \X)Y WHERE Y IS NOT cat-0
+
+%\def\trapactives#1{\trapactivechar{#1}\trapactivetok{#1}}
+%\def\trapactivechar#1{\tctestifnum{\number\catcode`#1=13}{\activechartrue}{}}
+%\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{%
+% \trapactivetokunexpandable#1}}
+%%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can at absorb@@
+%\def\trapactivetokunexpandable#1{\tctestifcon{\expandafter\if
+% \detokenize{#1}#1}{}{\activetoktrue\activetokunexpandabletrue\implicittoktrue}}
+% END REVISED ACTIVE TOK TRAP
+
+% ACTIVE IMPLICIT CAT-6 TRAP
+%\def\trapcatSIXc#1{\tctestifnum{\count at stringtoks{#1}>1}{\tc at defx\six at str{\string#1}%
+% \global\let\implicitsixtok\six at str\tc at tok{\implicitsixtok}}%
+% {\tc at tok\expandafter{\string#1}\tctestifnum{\number\catcode`#1=6}%
+% {}{\activetoktrue\implicittoktrue}}}
+% END ACTIVE IMPLICIT CAT-6 TRAP
+
+% ACTIVE CHAR \bgroup
+%\def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}%
+% {\expandafter\@@grpT\expandafter{\iffalse}\fi}% {
+% {\implicittoktrue\tc at implicitgrptrue%
+% \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED
+% {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup
+% {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup
+% \expandafter\@chrT\@tmp\@tokcycle}% ACTIVE CHAR \bgroup
+% }%
+%}
+% END ACTIVE CHAR \bgroup
+
+%%%%%%%%%%TRAPS ACTIVE-IMPLICIT GROUP TOKS FOR \stripimplicitgroupingcase
+%\def\implicitgrpfork#1{\tctestifcon{\iftc at implicitgrp}{\ifcase
+% \@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}{\@chrT#1}}
+
+%\def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}%
+% {\expandafter\@@grpT\expandafter{\iffalse}\fi}% {
+% {\implicittoktrue\tc at implicitgrptrue%
+% \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED
+% {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup
+% {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup
+% \expandafter\implicitgrpfork\@tmp\@tokcycle}% ACTIVE CHAR \bgroup
+% }}
+
+%\long\def\can at absorb@@#1{\let\@tmp#1\test at ifmacro\@tmp{\implicittokfalse
+% \@macT#1}{\trapimplicitegrp#1\implicitgrpfork#1}\@tokcycle}
+
+%\long\def\backslashcmds#1{%
+% \test at ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}%
+% {\implicittoktrue\trapimplicitegrp#1\implicitgrpfork#1}}
+%%%%%%%%%%END TRAPS ACTIVE-IMPLICIT GROUP TOKS FOR \stripimplicitgroupingcase
+
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -227,7 +318,7 @@
\begingroup
-\begin{exampleA}[lefthand width=7.8cm]{Employing \TokCycle{} directives}
+\begin{exampleA}[lefthand width=7.8cm]{}
\tokcycle{\addcytoks{\underdot{#1}}}
{\addcytoks{\{}\processtoks{#1}
\addcytoks{\}}}
@@ -239,7 +330,7 @@
\endgroup
-\subsubsection{Escaping text\label{s:esc}}
+\subsubsection{Escaping text}
Text between two successive escape characters is bypassed
by \TokCycle{} and instead echoed to the output register.
@@ -268,16 +359,15 @@
\endgroup
\subsubsection{Unexpandable, unexpanded, and expanded Character
- directives\label{s:unex}}
+ directives}
\begingroup
-These examples concern the issue of whether the characters of the input
+This section concerns the issue of whether the characters of the input
stream are transformed before or after being placed in the
output token register (\macname{cytoks}).\medskip
\noindent\begin{minipage}[t]{\textwidth}
-Transform characters (+1 ASCII) via unexpandable macro (see section~%
- \ref{s:esc} for \macname{plusl} definition):\medskip
+Transform characters (+1 ASCII) via unexpandable macro:\medskip
\begin{exampleA}[lefthand width=8.3cm]
{Unexpandable Character directive}
@@ -353,8 +443,6 @@
\Characterdirective{\addcytoks[x]{\vowelcap{#1}}}
\noindent\begin{minipage}[t]{\textwidth}
-See section~\ref{s:unex} for the \macname{vowelcap} definition.
-
\begin{exampleA}[lefthand width=5.5cm]{Normal token cycle (input stream
not pre-expanded)}
\tokcycle
@@ -410,11 +498,8 @@
implicit group tokens from the output; or $-$1 to instead pass the
implicit group tokens to the Character directive (as implicit
tokens) for separate processing (typically, when detokenization
- is desired).\smallskip
+ is desired).
-In the following example, see section~\ref{s:unex} for the
- \macname{vowelcap} definition.
-
\begin{exampleA}[lefthand width=6cm]{Using
\macname{stripimplicitgroupingcase} to affect treatment of
implicit grouping tokens}
@@ -438,6 +523,11 @@
\detokenize\expandafter{\the\cytoks}
\end{exampleA}
+%\tokcyclexpress{Announcement: {\bfseries\bgroup\itshape
+% Today \egroup it is} \today, a Wednesday.}\the\cytoks
+
+%\detokenize\expandafter{\the\cytoks}
+
\endgroup
\subsubsection{Treatment options for explicit groups}
@@ -529,7 +619,7 @@
\endgroup
-\subsubsection{Group nesting\label{s:nest}}
+\subsubsection{Group nesting}
\begingroup
@@ -562,10 +652,8 @@
\subsection{Direct use of \TokCycle}
-While one may find it convenient to encapsulate \TokCycle{} commands
- inside of other macros, it should be obvious that this is not a requirement.
-\TokCycle[T]{} macros and environments (in regular or \texttt{xpress}
- form) may be invoked directly from the document, without
+\TokCycle{} (in regular or \texttt{xpress} form) may be invoked
+ directly from the document, without
being first encapsulated within a macro or environment.
\subsubsection{Modifying counters as part of the Character
@@ -572,8 +660,6 @@
directive}
\begingroup
-In the following example, see section~\ref{s:nest} for the definitions
- of \macname{restorecolor} and \macname{reducecolor}.
\begin{exampleA}[lefthand width=8.5cm]{Using a period token (.) to
reset a changing color}
@@ -634,7 +720,7 @@
It is \textbf{not} to be confused as a replacement for
\macname{detokenize}.
In certain applications, it may offer a more pleasingly formatted
- typesetting of detokenized material.\smallskip
+ typesetting of detokenized material.
It is an unusual application of \TokCycle{} in that it does not
actually use the \macname{cytoks} token register to collect
@@ -880,14 +966,19 @@
\begingroup
Active characters in the \TokCycle{} input stream are processed in
their original form.
-Their active substitutions only occur \textit{afterwards}, when
+Their active substitutions arising from \macname{def}s
+ only occur \textit{afterwards}, when
the \TokCycle{} output is typeset.
-They may be identified in the Character directive with
- \macname{ifactivetok}.
-
+They may be identified with the \macname{ifactivetok} test.
+If \macname{let} to a character, they may be identified in the Character
+ directive;
+If \macname{let} to a control sequence or defined via \macname{def},
+ they may be identified in the Macro directive.
\let\svt T
\Characterdirective{\tctestifcon\ifactivetok
- {\addcytoks{\fbox{#1}}}{\addcytoks{#1}}}
+ {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}}
+\Macrodirective{\tctestifcon\ifactivetok
+ {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}}
\begin{exampleA}[lefthand width=7.8cm]{Processing active characters}
\resettokcycle
@@ -900,7 +991,9 @@
This is a test!!\endtokencyclexpress
\Characterdirective{\tctestifcon\ifactivetok
- {\addcytoks{\fbox{#1}}}{\addcytoks{#1}}}
+ {\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}}
+\Macrodirective{\tctestifcon\ifactivetok
+ {\addcytoks{\fbox{#1-mac}}}{\addcytoks{#1}}}
\tokencyclexpress
This is a test!!\endtokencyclexpress
@@ -913,24 +1006,29 @@
\end{exampleA}
%%%
+
+\ddbend If the input stream is pre-\textit{expanded}, any active
+ substitutions that are expandable (i.e., those involving \macname{def}
+ as well as those \macname{let} to something expandable)
+ are made before reaching \TokCycle{} processing.
+\svt hey are, thus, no longer detected as active, unless
+ \macname{noexpand} is applied before the pre-expansion.
+In this example, the \texttt{!} that is not \macname{noexpand}\kern.3pted
+ is converted to a \texttt{?} prior to reaching \TokCycle{}
+ processing (and thus, not detected as \macname{active}):
+
\catcode`!=\active
\def !{?}
\catcode`T=\active
\let T+
-\ddbend If the input stream is pre-\textit{expanded}, the active
- substitutions involving \macname{def} (but not \macname{let})
- are made before reaching {\catcode`T=11 \TokCycle} processing.
-Such tokens are no longer detected as active, unless
- \macname{noexpand} is applied to the pre-expansion:
-
\begin{exampleA}[lefthand width=7.8cm]{Expanded input stream acts
- upon active characters unless \macname{noexpand} is applied}
+ upon active \macname{def}ed characters unless \macname{noexpand} is applied}
\expandedtokcyclexpress{This is a test!\noexpand!}
\the\cytoks\par
\detokenize\expandafter{\the\cytoks}
\end{exampleA}
-However, pre-\textit{tokenization} does not suffer this behavior:
+However, pre-tokenization does not suffer this behavior:
\begin{exampleA}[lefthand width=7.8cm]{Pre-tokenized input stream
does not affect active characters}
\def\tmp{This is a test!!}
@@ -943,6 +1041,41 @@
\endgroup
\end{minipage}
+\begingroup
+\ddbend One aspect of \TeX{} to remember is that catcodes
+ are assigned at tokenization; however, for active characters,
+ the substitution assignment is evaluated only upon execution.
+So, if a cat-13 token is placed into a \macname{def}, it will
+ remain active even if the catcode of that character code is
+ later changed.
+But if the cat-13 active definition is changed prior to the
+ execution of the \macname{def}'ed token, the revised
+ token assignment will apply.
+
+The following example demonstrates this concept, while showing,
+ whithout changing the input in any way, that \TokCycle{}
+ can properly digest active and implicit grouping (cat-1,2) characters:
+
+\begin{exampleA}[lefthand width=8cm]{Active and implicit grouping
+ tokens digestible by \TokCycle}
+\catcode`Y=13
+\catcode`Z=13
+\let Y{
+\let Z}
+\let\Y{
+\let\Z}
+\def\tmp{\textit YabcZ de\Y\itshape f\Zg}%
+
+\def Y{\bgroup[NEW]}% APPLIES AT EXECUTION
+\catcode`Y=11% DOES NOT AFFECT Y IN \tmp
+
+\expandafter\tokcyclexpress\expandafter{\tmp}
+\the\cytoks
+
+\detokenize\expandafter{\the\cytoks}
+\end{exampleA}
+\endgroup
+
\subsubsection{Trap Catcode 6 (explicit \& implicit) tokens}
\begingroup
@@ -958,11 +1091,11 @@
into something cat-12 and set the if-condition \macname{catSIXtrue}.
In this manner, \macname{ifcatSIX} can be used inside the
Character directive to convert cat-6 tokens into something of
- the user's choosing.\smallskip
+ the user's choosing.
As to this cat-12 conversion, explicit cat-6 characters are converted
into the same character with cat-12.
-On the other hand, implicit cat-6 macros (e.g., \verb|\let\myhash#|)
+On the other hand, implicit cat-6 control sequences (e.g., \verb|\let\myhash#|)
are converted into a fixed-name macro, \macname{implicitsixtok},
whose cat-12 substitution text is a \macname{string} of the original
implicit-macro name.
@@ -1018,10 +1151,6 @@
In this example, a macro is defined and then executed, subject to
token replacements brought about by the expandable Character directive.
-\smallskip
-In the following example, see section~\ref{s:unex} for
- the \macname{vowelcap} definition.
-
\Characterdirective{%
\whennotprocessingparameter#1{\addcytoks[x]{\vowelcap{#1}}}}
\aftertokcycle{\global\cytoks\expandafter{\the\cytoks}}
@@ -1048,17 +1177,24 @@
\begingroup
-Implicit macros (assigned via \macname{let}) were already
- mentioned in the context of cat-6.
-However, implicit macros can be of any valid catcode.
+Implicit control sequences (assigned via \macname{let} to characters)
+ were already mentioned in the context of cat-6.
+However, implicit control sequences can be of any valid catcode (except
+ for cat-0, which we instead call macros or primitives).
The condition \macname{ifimplicittok} is used to flag such
- tokens for special processing.
+ tokens for special processing, as well as active tokens that
+ are \macname{let} to anything unexpandable.
+In the next example, implicit, cat-6 and implicit-cat-6 tokens may
+ all be differentiated, shown here with a multiplicity of
+ \macname{fbox}es.
+
\fboxsep=1.5pt
\begin{exampleA}[lefthand width=8cm]{Implicit\ensuremath{{}={}}%
single box\char44 \ cat-6\ensuremath{{}={}}double box\char44 \ %
implicit-cat-6\ensuremath{{}={}}triple box}
+\catcode`Q=\active \let QN
\let\littlet=t
\let\littlel=l
\let\svhash#
@@ -1073,10 +1209,48 @@
This \textit{is a \textbf{big}} \littlet est.
-Next pa#agraph ending with implicit cat six
+Qext pa#agraph ending with implicit cat six
\svhash.\endtokencyclexpress
\end{exampleA}
+In the following example, we use both control sequences and active characters
+ in \macname{def} and \macname{let} capacities, to demonstrate how
+ \TokCycle{} digests things.
+Implicit tokens (tokens \macname{let} to characters) are shown in a box,
+ with both the token name and the implicit value (note that tokens
+ \macname{let} to macros and primitives are not considered implicit).
+Active tokens processed through the character directive are followed with
+ a \dag, whereas those processed through the macro directive are followed
+ with a \ddag.
+
+
+\begin{exampleA}[lefthand width=7.5cm]{Non-active vs{.} active \macname{def}
+ \& \macname{let}}
+\Characterdirective{\ifimplicittok
+ \addcytoks{\fbox{\detokenize{#1}:#1}}%
+ \else\addcytoks{#1}\fi\ifactivetok
+ \addcytoks{\rlap{\dag}}\fi\addcytoks{~\,}}
+\Macrodirective{\ifimplicittok
+ \addcytoks{\fbox{\detokenize{#1}}}%
+ \else\addcytoks{#1}\fi\ifactivetok
+ \addcytoks{\rlap{\ddag}}\fi
+ \ifx\par#1\else\addcytoks{~\,}\fi}
+
+\def\A{a}
+\let\B i
+\let\C\today
+\let\D\relax
+\def\E{\relax}
+\catcode`V=13 \def V{a}
+\catcode`W=13 \let Ww
+\catcode`X=13 \let X\today
+\catcode`Y=13 \let Y\relax
+\catcode`Z=13 \def Z{\relax}
+\tokcyclexpress{\A\B\C\D\E ab\par VWXYZab}
+\the\cytoks
+\end{exampleA}
+
+
\ddbend If the input stream is subject to pre-expansion, one
will require \macname{noexpand} for macros where no pre-expansion
is desired.\medskip
@@ -1093,7 +1267,7 @@
Changing grouping tokens (catcodes 1,2) may require something more,
if the output stream is to be detokenized.
In the following examples, pay attention to the
- detokenized grouping around the argument to \macname{fbox}.\smallskip
+ detokenized grouping around the argument to \macname{fbox}.
As we will see, the issues raised here only affect the situation
when detokenization of the output stream is required.
Modified: trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex
===================================================================
--- trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex 2019-09-27 21:24:30 UTC (rev 52190)
+++ trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex 2019-09-27 21:24:43 UTC (rev 52191)
@@ -1,7 +1,7 @@
\def\tcname {tokcycle}
-\def\tcver {1.0}
+\def\tcver {1.1}
%
-\def\tcdate {2019/8/21}
+\def\tcdate {2019/9/27}
%
% Author : Steven B Segletes, Christian Tellechea (contributor)
% Maintainer : Steven B Segletes
@@ -52,13 +52,15 @@
\long\def\tc at exfirst#1#2{#1}
\long\def\tc at exsecond#1#2{#2}
\long\def\tc at testxifx{\tc at earg\tctestifx}
+\long\def\test at ifmacro#1{\tctestifcatnx#1\relax}
\def\tc at addtoks#1#2{\toks#1\expandafter{\the\toks#1#2}}
\def\addtc at depth{\advance\tc at depth 1}
\def\subtc at depth{\tc at depth=\numexpr\tc at depth-1\relax}
-\def\tc at resetifs{\activetokfalse\implicittokfalse\tc at implicitgrpfalse\catSIXfalse}
+\def\tc at resetifs{\activetokfalse\implicittokfalse\tc at implicitgrpfalse
+ \catSIXfalse\activecharfalse\activetokunexpandablefalse}
\long\def\count at stringtoks#1{\tc at earg\count at toks{\string#1}}
-\def\count at toks#1{\the\numexpr-1\count@@toks#1.\tc at endcnt}
-\def\count@@toks#1#2\tc at endcnt{+1\tc at ifempty{#2}{\relax}{\count@@toks#2\tc at endcnt}}
+\long\def\count at toks#1{\the\numexpr-1\count@@toks#1.\tc at endcnt}
+\long\def\count@@toks#1#2\tc at endcnt{+1\tc at ifempty{#2}{\relax}{\count@@toks#2\tc at endcnt}}
\def\sv at hash{##}
% EXTERNAL TOOLS
\long\def\tctestifcon#1{#1\expandafter\tc at exfirst\else\expandafter\tc at exsecond\fi}
@@ -70,6 +72,8 @@
\newif\ifcatSIX
\newif\ifimplicittok
\newif\ifactivetok
+\newif\ifactivechar
+\newif\ifactivetokunexpandable
\newtoks\cytoks
\long\def\tokcycleenvironment#1#2#3#4#5{\expandafter\def\expandafter#1%
\expandafter{\expandafter\let\csname end\expandafter\tc at gobble
@@ -89,14 +93,15 @@
% LOOP ENTRY POINT
\def\@tokcycle{\tc at resetifs\futurelet\tc at next\detect at CANTabsorb}
\def\detect at CANTabsorb{\tctestifx{\tc at next\tc at sptoken}{\stringify\@@@@@spcT}%
- {\tctestifx{\tc at next\bgroup}{\stringify\@@@@grpT}%
- {\can at absorb}}}
+ {\tctestifx{\tc at next\bgroup}{\stringify\@@@@grpT}{\can at absorb}}}
% NON cat1,10 TOKENS
\long\def\can at absorb#1{\tc at tok{#1}\trapcatSIX{#1}\expandafter\can at absorb@
\the\tc at tok}
\long\def\can at absorb@#1{\tctestifnum{\count at stringtoks{#1}>1}%
{\tctestifx{\endtokcycraw#1}{#1}{\backslashcmds#1\@tokcycle}}%
- {\trapactivetok#1\tc at trapescape#1{\tc at escapecytoks}{\@chrT#1\@tokcycle}}}
+ {\trapactives#1\tc at trapescape#1{\tc at escapecytoks}{\can at absorb@@#1}}}
+\long\def\can at absorb@@#1{\let\@tmp#1\test at ifmacro\@tmp{\implicittokfalse
+ \@macT#1}{\trapimplicitegrp#1\implicitgrpfork#1}\@tokcycle}
%CONVERT NEXT (SPACE OR BEGIN-GROUP) TOKEN TO STRING
\def\stringify#1{\expandafter#1\string}% #1 WILL BE \@@@@@spcT or \@@@@grpT
%SPACE DECODE
@@ -110,17 +115,21 @@
\def\@@@@grpT{\futurelet\tc at str\@@@grpT}
\def\@@@grpT#1{\tctestifnum{\number\catcode`#1=1}%
{\expandafter\@@grpT\expandafter{\iffalse}\fi}% {
- {\implicittoktrue\tc at implicitgrptrue
- \csmk{\expandafter\backslashcmds\thecs\@tokcycle}}}% \bgroup
+ {\implicittoktrue\tc at implicitgrptrue%
+ \tctestifnum{`#1=92}% WORKS EVEN IF CAT-0 HAS CHANGED
+ {\csmk{\expandafter\backslashcmds\thecs\@tokcycle}}% \bgroup
+ {\begingroup\catcode`#1=\active \xdef\@tmp{\scantokens{#1\noexpand}}\endgroup
+ \expandafter\implicitgrpfork\@tmp\@tokcycle}% ACTIVE CHAR \bgroup
+ }}
\def\@@grpT#1{\tctestifcon\ifstripgrouping{\@grpT{#1}}%
{\groupedcytoks{\@grpT{#1}}}\@tokcycle}
% \ COMMANDS (MACROS AND IMPLICITS)
-\long\def\test at ifmacro#1{\tctestifcatnx#1\relax}% REVISED!
-\long\def\backslashcmds#1{\test at ifmacro#1{\tctestifcon\ifcatSIX{%
- \implicittoktrue\@chrT#1}{\@macT#1}}{\implicittoktrue
- \trapimplicitegrp#1\tctestifcon{\iftc at implicitgrp}%
- {\ifcase\@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}%
- {\@chrT#1}{\@chrT#1}}}
+\long\def\backslashcmds#1{%
+ \test at ifmacro#1{\tctestifcon\ifcatSIX{\implicittoktrue\@chrT#1}{\@macT#1}}%
+ {\implicittoktrue\trapimplicitegrp#1\implicitgrpfork#1}}
+% FORK BASED ON IMPLICIT GROUP TREATMENT
+\def\implicitgrpfork#1{\tctestifcon{\iftc at implicitgrp}{\ifcase
+ \@implicitgroupingcase\or\addcytoks{#1}\or\@chrT{#1}\fi}{\@chrT#1}}
% SET UP ESCAPE MECHANISM
\def\settcEscapechar#1{\let\@tcEscapeptr#1%
\def\tc at escapecytoks##1#1{\addcytoks{##1}\@tokcycle}}
@@ -130,7 +139,8 @@
\def\trapcatSIXb#1{\tc at earg\tctestifcatnx\sv at hash#1{\catSIXtrue\trapcatSIXc#1}{}}
\def\trapcatSIXc#1{\tctestifnum{\count at stringtoks{#1}>1}{\tc at defx\six at str{\string#1}%
\global\let\implicitsixtok\six at str\tc at tok{\implicitsixtok}}%
- {\tc at tok\expandafter{\string#1}}}
+ {\tc at tok\expandafter{\string#1}\tctestifnum{\number\catcode`#1=6}%
+ {}{\activetoktrue\implicittoktrue}}}
% DIRECTIVES FOR HANDLING GROUPS RECURSIVELY; DEFINE tokcycle GROUPING CHARS
\def\@defgroupedcytoks#1{\long\def\groupedcytoks##1{%
\begingroup\cytoks{}##1\expandafter\endgroup\expandafter
@@ -154,7 +164,12 @@
\def\trapimplicitegrp#1{\tctestifx{#1\egroup}{%
\implicittoktrue\tc at implicitgrptrue}{}}
% TRAP ACTIVE TOK
-\def\trapactivetok#1{\tctestifnum{\number\catcode`#1=13}{\activetoktrue}{}}
+\def\trapactives#1{\trapactivechar{#1}\trapactivetok{#1}}
+\def\trapactivechar#1{\tctestifnum{\number\catcode`#1=13}{\activechartrue}{}}
+\def\trapactivetok#1{\tctestifcatnx~#1{\activetoktrue}{\trapactivetokunexpandable#1}}
+%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can at absorb@@
+\def\trapactivetokunexpandable#1{\tctestifcon{\expandafter\if
+ \detokenize{#1}#1}{}{\activetoktrue\activetokunexpandabletrue\implicittoktrue}}
% EXPRESS-INTERFACE - ALLOWS TO EXTERNALLY DEFINE DIRECTIVES
\def\Characterdirective{\def\@chrT##1}
\def\Groupdirective{\long\def\@grpT##1}
@@ -190,3 +205,12 @@
EDIT HISTORY
v1.0 2019/8/21
- Initial release
+
+v1.1 2019/9/27
+ - Introduced \ifactivechar, \ifactivetokunexpandable
+ - Tightened up consistent definition of implicit (to exclude primitives)
+ - Rewrote active token trapping logic, to differentiate between active
+ token vs. active character code, in the event that an earlier tokenized
+ token no longer shares the current characteristics of the character code
+ - Added ability to handle active-implicit grouping tokens
+ - Added ability to handle active-implicit cat-6 tokens
\ No newline at end of file
More information about the tex-live-commits
mailing list