texlive[59377] Master/texmf-dist: tokcycle (28may21)
commits+karl at tug.org
commits+karl at tug.org
Fri May 28 22:48:44 CEST 2021
Revision: 59377
http://tug.org/svn/texlive?view=revision&revision=59377
Author: karl
Date: 2021-05-28 22:48:44 +0200 (Fri, 28 May 2021)
Log Message:
-----------
tokcycle (28may21)
Modified Paths:
--------------
trunk/Master/texmf-dist/doc/generic/tokcycle/README
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/README
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/README 2021-05-28 20:48:24 UTC (rev 59376)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/README 2021-05-28 20:48:44 UTC (rev 59377)
@@ -5,7 +5,7 @@
Files : 1) tokcycle.tex (tex/generic/tokcycle)
2) tokcycle.sty (tex/latex/tokcycle)
- 3) tokcycle-doc.tex
+ 3) tokcycle-doc.tex (doc/latex/tokcycle)
4) tokcycle-doc.pdf (doc/latex/tokcycle)
5) tokcycle-examples.tex (doc/latex/tokcycle)
6) tokcycle-examples.pdf (doc/latex/tokcycle)
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
===================================================================
(Binary files differ)
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex 2021-05-28 20:48:24 UTC (rev 59376)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex 2021-05-28 20:48:44 UTC (rev 59377)
@@ -25,7 +25,12 @@
\newcommand\Space{\textit{Space}}
\newcommand\CGMS{\textit{C-G-M-S}}
\newcommand\CGMnS{\textit{C}, \textit{G}, \textit{M}, and \textit{S}}
-\newcommand\macname[1]{\texttt{\char92#1}}
+\begingroup
+\catcode`|=\active
+\gdef|{\textbackslash}
+\gdef\xmacaux#1{\ttfamily\char92#1\endgroup}
+\endgroup
+\gdef\macname{\begingroup\catcode`|=\active \xmacaux}
\newcommand\thevbox{\medskip\theverbbox\par\medskip\noindent}
\newenvironment{specfig}
{\par\bigskip\vbox\bgroup\small\sffamily\centering}
@@ -153,7 +158,7 @@
with that category of token/group.
The \TokCycle{} categories by which the input stream
is dissected include \Character{},
- \Group{}, \Macro{}, and \Space{}.
+ \Group{}, \Macro{}, and \Space{} (alphabetized for easy recall).
Catcode-0 tokens are directed to the \Macro{} directive for processing.
Catcode-10 tokens are directed to the \Space{} directive.
@@ -170,6 +175,8 @@
Valid tokens that are neither catcode 0, 1, 2, nor 10, except where noted,
are directed to the \Character{} directive for processing.
+\subsection{Provided (built-in) \TokCycle{} macros and environments}
+
The syntax of the non-\texttt{xpress} macros/environments is\medskip
\begin{verbbox}[\vbdelim]
@@ -206,6 +213,8 @@
\thevbox
\begin{sloppypar}
+\subsection{Create your own \TokCycle{} environments}
+
In addition to the above macros/environments, the means is
provided to define new \TokCycle{} environments:
@@ -311,7 +320,7 @@
\section{Commands in the \TokCycle{} directives}\label{s:coms}
-The command-line token cycling tools provided in the package are
+The document-level token cycling tools provided in the package are
listed in section~\ref{s:intro}.
For each of those commands and/or pseudo-environments, the
user must (explicitly or implicitly) detail a set of directives to
@@ -326,7 +335,12 @@
The recommended way to apply this package is to collect the
\TokCycle-transformed results of the input stream in a token register
provided by the package, named \macname{cytoks}.
-Its contents can then be typeset via \verb|\the\cytoks|.
+Its contents can then be typeset via \verb|\the\cytoks|.\footnote{%
+If a token-cycle input stream contains no macros, or is to be detokenized,
+ or if the input-stream tokens are not to be typeset, it may be possible
+ (though not required) to bypass \macname{cytoks} and typeset the output
+ directly.
+}
The macro for appending things to \macname{cytoks}, to be used in
the package directives, is \macname{addcytoks}.
@@ -365,7 +379,7 @@
in such a way that ``\argone{}'' may be directly employed to indicate
the current token (or group) under consideration in the input stream.
-\subsubsection{Transforming the Input Stream}
+\subsubsection{Transforming the input stream}
Within the \TokCycle{} \CGMS{} directives, the command
\verb|\addcytoks{#1}| is used to echo the token being processed to
@@ -449,8 +463,8 @@
(for example, one that might add additional tokens to \macname{cytoks})
immediately \textit{outside} of the group, before or after.
If you were to add the task macro before or after the \macname{processtoks}
- invocation, it will still be performed, by default, \textit{inside}
- the \macname{cytoks} group.
+ invocation (anywhere in the \Group{} directive), it will still be
+ performed, by default, \textit{inside} the \macname{cytoks} group.
The macro \macname{groupedcytoks} allows one to manually specify the
grouping duties of the \Group{} directive.
@@ -518,6 +532,201 @@
Such a special treatment has limited application---for example,
when the intent is to detokenize these tokens.
+\subsection{Looking ahead at the input stream}
+
+In the normal mode of \TokCycle{} operation, each token of the input stream
+ is successively digested and sent to one of four user-defined directives
+ for processing.
+Such an approach works well if there is no interdependency of adjacent tokens
+ in the input stream.
+When such an interdepency exists (for example, the argument associated with
+ an invoked macro), one might typically use flags that are set or cleared
+ in one directive that can then be status-checked when the subsequent token
+ is processed.
+While such an approach is wholly valid, it can create a complex web of
+ flag setting/checking that can span across multiple directives.
+
+With the release of v1.4 of the package, an alternative approach to handle
+ such dependencies has been developed: look-ahead features, so that the
+ occurrence of a particular token or condition can provoke an immediate
+ examination of the input stream to handle a possible dependency, without
+ waiting for the next iteration of the token cycle.
+
+It should be noted that the use of look-ahead is perhaps more
+ \textit{dangerous} than the traditional use of flags---for if it is not
+ performed with care, tampering with the input stream can destroy its
+ integrity.
+The commands developed for these situations will now be described.
+They are designed to be employed as part of \TokCycle's \Character, \Macro,
+ and \Space{} directives for checking and/or handling successively linked
+ tokens in the input stream.
+They cannot be used inside the \Group{} directive.
+
+\subsubsection{\macname{tcpeek}}
+
+When issued witin a directive as, for example \macname{tcpeek|z}, the next
+ token from the input stream is \macname{futurelet} into \macname{z}.
+This has several implications.
+The input stream remains undisturbed, so that the peeked-at token will
+ still be the token to be digested in the next iteration of the token cycle.
+Because it has been \macname{let}, \macname{z} does not \textit{contain} the token
+ from the input stream in the manner of a \macname{def}, but rather it is a
+ separate token \textit{possessing the same properties} as the input-stream token.
+As such, it is ideal to be used for \macname{ifx} comparisons.
+For example, \macname{ifx|bgroup|z} will detect whether a cat-1 group-opening
+ token is at the head of the input stream, e.g., an opening brace, `\{';
+the comparison \macname{ifx|tcsptoken|z} will detect whether a cat-10 space
+ is at the head of the input stream.
+Such comparisons can be useful in directing the logic of the \TokCycle{}
+ directive.
+
+\subsubsection{\macname{tcpop}}
+
+When issued as, for example \macname{tcpop|z}, the next token(s) from the
+ input stream is \textit{immediately} digested as an argument and the
+ tokens are placed into the macro \macname{z}.
+This token (or tokens) will no longer be digested as part of the next
+ iteration within the token cycle, unless the tokens are subsequently
+ replaced at the head of the input stream (see \macname{tcpush} below).
+
+Beware that when \TeX{} absorbs an argument, leading white space is lost.
+Further, if the argument was enclosed in cat-1,2 braces, the braces are
+ stripped and the whole group is absorbed as the argument.
+In some cases, if the argument needs manipulation, this brace-stripping
+ behavior may be desired.
+However, if the retention of the leading white space and the braces are
+ desired, one can use manual techniques (possibly involving macros
+ such as \macname{tcpeek}, \macname{tcpopwhitespace} and
+ \macname{tcpushgroup}) or one can instead use \macname{tcpopliteral}
+ (see below) in lieu of \macname{tcpop}.
+
+If \macname{tcpop} is used nonetheless, to aid in such matters when
+ cat-10 space is at the head of the input stream, the flag
+ \macname{spacepoppedtrue} will be set when a \macname{tcpop} is issued.
+However, even \macname{ifspacepopped} will be unable to differentiate explicit
+ ``white'' space that is lost during argument absorption versus implicit
+ space (e.g., \macname{@sptoken} or active-implicit space) that is not
+ discarded.
+Another option for dealing with leading white space is
+ \macname{tcpopwhitespace} (see below).
+
+
+\subsubsection{\macname{tcpopliteral}}
+
+This macro is an alternative to \macname{tcpop}, if the retention of
+ leading white space and possibly braces are required when popping
+ an argument from the input stream.
+This can be particularly useful if the popped tokens must later be
+ placed back into the input stream in their original state.
+The syntax, \macname{tcpopliteral|z}, pops an argument into \macname{z},
+ while retaining possible leading whitespace and brace groups.
+
+\subsubsection{\macname{tcpopappto}\ and \macname{tcpopliteralappto}}
+
+When issued as, for example \macname{tcpopappto|z}, the next token(s) from the
+ input stream is absorbed as an argument and \textit{appended} to the
+ replacement text of \macname{z}.
+The same group/space provisos affecting \macname{tcpop} apply here as well.
+The control sequence to be appended, here \macname{z}, may not be undefined
+ when \macname{tcpopappto} is invoked.
+This macro is just a convenient joining of two macros: \macname{tcpop}
+ and \macname{tcappto\#1from\#2} (see below).
+
+The same applies for the macro \macname{tcpopliteralappto}, which conveniently
+ joins \macname{tcpopliteral} and \macname{tcappto\#1from\#2}.
+
+\subsubsection{\macname{tcpopuntil}}
+
+This command pops one or more tokens from the input stream in the manner of
+ a delimited argument.
+Thus, when issued as, for example \macname{tcpopuntil~0|z}, tokens from
+ the input stream are absorbed into \macname{z} until an \texttt{0} token is
+ reached.
+Unlike delimited-argument absorption, however, the delimiter (\texttt{0} in
+ this example) is also added to the specified control sequence, in this case
+ \macname{z}.
+This construct is very useful for obtaining optional-argument tokens from the
+ input stream.
+Consider the following code, with an input stream of \texttt{[1ex]\{2ex\}}:
+\begin{verbatim}
+ \tcpeek\z
+ \ifx[\z\tcpopuntil ]\q\else\def\q{}\fi
+\end{verbatim}
+Since an optional argument is next in the input stream, \macname{q} will
+ obtain the replacement text \texttt{[1ex]}, and the input stream will now
+ begin with \texttt{\{2ex\}}.
+If an optional argument had not been next in the input stream, then
+ \macname{q} would be empty and the input stream would still begin
+ with \texttt{\{2ex\}}.
+
+\subsubsection{\macname{tcpopwhitespace}}
+
+When issued as, for example \macname{tcpopwhitespace|z}, leading white space
+ from the input stream will be absorbed and set in \macname{z}, in the
+ manner of a \macname{def}.
+Unlike the flag \macname{ifspacepopped}, leading implicit space will not be
+ indicated by this macro---only leading explicit white space.
+If no leading white space is present, \macname{z} will be empty following
+ its invocation and no tokens will have been removed from the input stream.
+
+\subsubsection{\macname{tcpush}}
+
+To this point, several commands have been described for reading and/or
+ extracting tokens from the input stream.
+There are also a corresponding commands for placing material at the head
+ of the input stream, should that need arise.
+With an invocation of \macname{tcpush|z}, the replacement text of \macname{z}
+ will be pushed onto the input stream.
+Multiple pushes will be handled in a last-in-first-out fashion.
+
+Beware that, because argument absorption in \TeX{} will strip the braces of
+ an absorbed group, care must be taken. If the input stream leads with a
+ grouped quantity, such as \texttt{\{abc\}}, then \macname{tcpop|z|tcpush|z}
+ will end up with \texttt{abc} in the input stream, with the braces missing.
+\textit{The commands \macname{tcpop} and \macname{tcpush} are not strictly
+ inverse operations} (see \macname{tcpopliteral} as an alternative).
+For this reason, the command \macname{tcpushgroup} is also provided
+ (see below).
+
+The command \macname{tcpush} supports an optional argument that functions
+ in the same manner as that of \macname{addcytoks}, to provide additional
+ levels of expansion beyond the mere replacement text.
+Thus, \macname{tcpush[2]|z} will take the replacement text of \macname{z},
+ expand it twice, and push the result onto the input stream.
+In a similar way, \macname{tcpush[x]|z} will fully expand the replacement
+ text of \macname{z} and then place those tokens onto the input stream.
+Whereas \macname{addytoks} places tokens into the output stream (the
+ token list \macname{cytoks}), \macname{tcpush} places tokens at the
+ head of the input stream.
+
+\subsubsection{\macname{tcpushgroup}}
+
+The command \macname{tcpushgroup} functions in a manner similar to
+ \macname{tcpush}, except that the replacement text of the argument
+ is placed onto the input stream within a braced (cat-1,2) group.
+So if the replacement text of \macname{z} is \texttt{abc}, then the
+ invocation \macname{tcpushgroup|z} will place \texttt{\{abc\}} onto
+ the input stream.
+As with \macname{tcpush}, and with the identical syntax, the command
+ \macname{tcpushgroup} supports an optional argument that directs
+ addition levels of expansion beyond the mere replacement text.
+
+\subsubsection{\macname{tcappto\#1from\#2}}
+
+This macro does not touch the input stream, per se.
+However, as tokens are popped from the input stream and placed into macros,
+ this command conveniently allows for their aggregation
+Thus, if \macname{q} contains \texttt{abc} and the newly popped \macname{z}
+ contains \texttt{d}, then \macname{tcappto |q from |z} will append the
+ contents of \macname{z} to \macname{q}, so that, upon conclusion,
+ \macname{q} will contain the tokens \texttt{abcd}.
+Unexpandable tokens can also be appended directly using this macro, using
+ the syntax of \macname{tcappto |q from\{123\}}.
+Expandable tokens can also be added directly with the use of this macro,
+ by using a leading \macname{noexpand} or \macname{empty}, as in
+ \macname{tcappto |q from\{|noexpand|today\}}.
+
+
\subsection{Truncating the input stream}
The basic process of \TokCycle{} is one in which an input stream (or argument)
@@ -554,7 +763,7 @@
or for the remainder of the total \TokCycle{}
input stream.
-\subsubsection{\macname{truncategroup} and \macname{truncategroupiftokis}}
+\subsubsection{\macname{truncategroup}\ and \macname{truncategroupiftokis}}
\label{s:truncgrp}
As to truncating the input stream for the remainder of the \TokCycle{}
@@ -596,7 +805,7 @@
Obviously, checks for particular character tokens would be placed in the
\Character{} directive, rather than the \Macro{} directive.
-\subsubsection{\macname{truncatecycle} and \macname{truncatecycleiftokis}}
+\subsubsection{\macname{truncatecycle}\ and \macname{truncatecycleiftokis}}
There are two commands that are in every way analogous to the
group-\hspace{0pt}terminating macros of section~\ref{s:truncgrp}.
@@ -746,7 +955,7 @@
ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed
by the \Space{} directive.
-\subsubsection{Active-Implicit Tokens, Including Spaces}
+\subsubsection{Active-implicit tokens, including spaces\label{s:acimsp}}
One may occasionally run across a token that is both active and implicit.
For example, in the following code, \texttt{Q} is made both active and implicit:
@@ -761,18 +970,20 @@
This is true even in the case of active-implicit catcode-10 spaces, which
are always processed through the \Space{} directive.
-However, because of the process \TokCycle{} uses in digesting space
- tokens, the actual token passed to the \Space{} directive (so-called
- \verb|#1|), when an active-implicit space is encountered in the input
- stream, is a generic implicit space token named \macname{tc at sptoken}.
-However, the catcode-12 version of the active character that
- produced it will be, for that moment, retained in a definition
+As of \TokCycle{} v1.4, the actual active-implicit space, if
+ encountered in the input stream, will be passed as \verb|#1| to
+ the \Space{} directive, \textit{as long as the character code of the
+ token is still \macname{active}.}
+If the character code of that token is no longer active, a generic
+ implicit space token named \macname{tcsptoken} is instead passed
+ to the the \Space{} directive as \verb|#1|.
+In either case, the catcode-12 version of the active character that
+ was digested will be, for that moment, retained in a definition
named \macname{theactivespace}.
This can be useful if detokenization is required of the spaces.
Such an example is described in the \texttt{tokcycle-examples}
adjunct document.
-
\subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}}
\label{s:cat6}
@@ -846,7 +1057,7 @@
catcodes of the two input toks \textbf{without expansion} (\texttt{nx}
denotes \macname{noexpand}).
-\subsection{\TokCycle{} Completion: \macname{aftertokcycle}
+\subsection{\TokCycle{} completion: \macname{aftertokcycle}\
and \macname{tcendgroup}}
The \TokCycle{} \textit{macros}, upon completion, do nothing.
@@ -957,12 +1168,18 @@
\item A similar warning is to be made for active-implicit spaces as was
made for implicit-cat-6 tokens.
-In the case of active-implicit spaces, the name of the active character is
- momentarily stored in the \macname{theactivespace} macro.
-If one wishes to output the active \textit{name} to the \macname{cytoks} token list,
- one must make sure that \macname{theactivespace} is expanded once before
+In the case of active-implicit spaces, the name (a cat-12 string) of the
+ active character is momentarily stored in the \macname{theactivespace} macro.
+If one wishes to output the active \textit{name} to the \macname{cytoks} token
+ list, one must make sure that \macname{theactivespace} is expanded once before
being added to the token list.
+One difference here, however, is that, unlike a cat-6 token, an
+ active-implicit space (and not a string of it) is actually passed
+ to the \Space{} directive as \texttt{\#1}.
+In all but one special case, the active-implicit \texttt{\#1} will match that
+ which occurred in the input stream (see section~\ref{s:acimsp}).
+
\end{itemize}
\section*{Acknowledgments}
@@ -980,13 +1197,13 @@
opportunity to interact with the leading developers and
practitioners of \TeX{} and \LaTeX{}.
-\vspace{1in}
+\vspace{.6in}
\section*{Source Code}
%\vspace{-0.1in}
\Large\texttt{tokcycle.sty}
\verbfilenobox[\footnotesize]{tokcycle.sty}%
-%\vspace{-0.1in}
+\vspace{1.5in}
\noindent\texttt{tokcycle.tex}
\verbfilenobox[\footnotesize]{tokcycle.tex}
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
===================================================================
(Binary files differ)
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex 2021-05-28 20:48:24 UTC (rev 59376)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex 2021-05-28 20:48:44 UTC (rev 59377)
@@ -1,7 +1,11 @@
\documentclass[10pt]{article}
\usepackage[margin=1.3in]{geometry}
\usepackage[T1]{fontenc}
-\usepackage{tokcycle,xcolor,manfnt,lmodern}
+\usepackage{tokcycle,xcolor,manfnt,lmodern,mathtools}
+%%%%TRIAL PACKAGE CHANGES (MAKE SURE EXAMPLE FILE COMPILES CORRECTLY)
+\makeatletter
+\makeatother
+%%%%END TRIAL
\colorlet{bred}{red!90!blue}
\usepackage{tikz}
% MACROS: REDEFINE TO ALLOW \cytoks TO ESCAPE SCOPE OF exampleA
@@ -48,7 +52,8 @@
\def\altdecytoks{\par\medskip\begingroup\noindent\macname{cytoks}
\ \textit{alt}detokenization:\\\raggedright\footnotesize\ttfamily
- \expandafter\altdetokenize\expandafter{\the\cytoks}\normalsize\par\endgroup}
+ \expandafter\altdetokenize\expandafter{\the\cytoks}\normalsize\par
+ \endgroup\medskip}
\newcommand\TokCycle{\textsf{tokcycle}}
@@ -120,21 +125,21 @@
\newcommand\nextcap[1]{%
\edef\tmp{#1}%
- \tctestifx{-#1}{\def\capnext{T}}{}%
+ \tctestifx{-#1}{\def\capnext{T}}{}% <- TEST FOR HYPHEN
\tctestifcon{\if T\capnext}%
- {\tctestifcon{\ifcat\tmp A}%
- {\uppercase{#1}\def\capnext{F}}%
- {#1}}%
- {#1}%
+ {\tctestifcon{\ifcat\tmp A}% <- TEST FOR NEXT CAT-11
+ {\addcytoks{\uppercase{#1}}\def\capnext{F}}%
+ {\addcytoks{#1}}}%
+ {\addcytoks{#1}}%
}
\newcommand\TitleCase[1]{%
- \def\capnext{T}
+ \def\capnext{T}% <- INITIAL ENTRY
\tokcycle
- {\addcytoks{\nextcap{##1}}}
- {\processtoks{##1}}
+ {\nextcap{##1}}
+ {\def\capnext{T}\processtoks{##1}}% <- GROUP ENTRY
{\addcytoks{##1}}
- {\addcytoks{##1\def\capnext{T}}}
+ {\addcytoks{##1}\def\capnext{T}}% <-CAT-10 SPACE
{#1}%
\the\cytoks
}
@@ -201,24 +206,23 @@
\newcommand\nextcmptok[1]{\rotoradd{\rotcytoks{#1}}{\addcytoks{#1}}\testmatch{\tmp}}
\newcommand\nextctltok[1]{\stepcounter{runcount}\addcytoks{#1}}
-\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}%
- {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}}
-
\newcounter{sigd}
\newcounter{trailingz}
+\newif\iffounddot
+\newif\ifzerosig
\newcommand\sigdigits[1]{%
\setcounter{sigd}{0}% SIGNIFICANT DIGIT COUNT
\setcounter{trailingz}{0}% TRAILING ZERO COUNT
- \def\founddot{F}% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED
- \def\zerosig{F}% MADE T WHEN ZERO BECOMES SIGNIFICANT
+ \founddotfalse% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED
+ \zerosigfalse% MADE T WHEN ZERO BECOMES SIGNIFICANT
\tokcycle% CYCLE THROOUGH EACH TOKEN
{\tctestifx{.##1}%
- {\if T\founddot\throwerr\fi\def\founddot{T}}% IF .
+ {\iffounddot\throwerr\fi\founddottrue}% IF .
{\tctestifnum{`##1>`/}%
{\tctestifnum{`##1<`:}%
{\tctestifnum{##1>0}%
- {\stepcounter{sigd}\def\zerosig{T}\setcounter{trailingz}{0}}% IF 1-9
- {\if T\zerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0
+ {\stepcounter{sigd}\zerosigtrue\setcounter{trailingz}{0}}% IF 1-9
+ {\ifzerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0
}%
{\throwerr}% IF ASCII > `9
}%
@@ -229,7 +233,7 @@
{\throwerr}% IF CONTROL SEQUENCE
{}% IGNORE SPACES
{#1}% THE ARGUMENT
- \addtocounter{sigd}{-\thetrailingz}% SUBTRACT TRAILING ZEROS
+ \iffounddot\else\addtocounter{sigd}{-\thetrailingz}\fi% TRAILING ZEROS
\expandafter\ifx\expandafter\relax\detokenize\expandafter{\the\cytoks}%
\relax\else\setcounter{sigd}{-1}\fi% CHECK FOR INVALID INPUT
\thesigd
@@ -269,7 +273,82 @@
{\dumpword\ifrunningcount\else\tcafterenv{\par(Wordcount=\thewordcount,
Lettercount=\thelettercount)}\fi}
+\newcounter{pointcount}
+\newcommand\addtopoints[1]{(\textit{#1 point}%
+ \ifnum#1=1\relax\else\textit{s}\fi)}
+\NewEnviron{pointtracker}[2][points]{%
+ \par\bigskip\resettokcycle
+ \setcounter{pointcount}{0}%
+ \Macrodirective{\addcytoks{##1}\tctestifx{\addtopoints##1}{\sumpoints}{}}%
+ \def\tmp{\tokencyclexpress{\large\bfseries#2: \thepointcount\ #1}}%
+ \expandafter\tmp\BODY\endtokencyclexpress
+}
+\def\sumpoints{\tcpop\Q\addtocounter{pointcount}{\Q}\tcpushgroup{\Q}}
+\newcommand\scalerule[1]{%
+ \tcpeek\zA
+ \ifx[\zA\tcpopuntil]\opt\fi% IF PRESENT, POP OPTIONAL ARG
+ \tcpop\ArgA% POP ARG1
+ \tcpop\ArgB% POP ARG2
+ \tcpushgroup[x]{#1\dimexpr\ArgB}% PUSH BACK IN REVERSE (LIFO) FASHION
+ \tcpushgroup[x]{#1\dimexpr\ArgA}
+ \ifx[\zA\tcpush[x]{\expandafter\scaleopt\opt{#1}}\fi
+}
+\def\scaleopt[#1]#2{[#2\dimexpr#1]}
+
+\newcommand\protectruleargs[1]{\ifx\rule#1%
+ \tcpeek\zA
+ \ifx[\zA\tcpopuntil]\opt
+ \addcytoks[1]{\opt}\fi% IF OPTIONAL ARG PRESENT, POP AND SAVE TO \cytoks
+ \tcpopliteral\ArgA% MUST POP ARG 1 WITH BRACES
+ \addcytoks[1]{\ArgA}%
+ \tcpopliteral\ArgB% MUST POP ARG 2 WITH BRACES
+ \addcytoks[1]{\ArgB}%
+ \fi
+}
+
+\newif\iffoundstring
+\newcommand\findstring[4]{% {CURRENT TOK}{MATCH STRING}{SUCCESS OP}{FAIL CMDS}
+ \foundstringfalse% DEFAULT CONDITION
+ \def\poptoks{}% INITIALIZATION. IF MATCH FAILS, MUST TO RESTORE POPPED TOKENS
+ \expandafter\matchnextchar\ifx#1#2\relax\relax\fi\empty\relax
+ \iffoundstring\addcytoks{#3{#2}}\else\tcpush{\poptoks}#4\fi
+}
+\def\matchnextchar#1#2\relax{\tctestifx{#1\empty}{}{% NO MATCH FROM 1ST TOKEN
+ \tctestifx{\relax#1}{%
+ \foundstringtrue% NOTHING LEFT TO COMPARE; MATCH FOUND
+ }{%
+ \tcpeek\Q% WITHOUT POPPING...
+ \tctestifx{\Q\bgroup}{}{% PEEK FOR GROUPS OR...
+ \tctestifx{\Q\tcsptoken}{}{% SPACES THAT INDICATE MATCH FAILURE
+ \tcpopappto\poptoks% POP TOKEN AND APPEND TO \poptoks
+ \expandafter\tctestifx\expandafter{\Q#1}{%
+ \matchnextchar#2\relax% STILL MATCHING; KEEP GOING
+ }{}%
+ }%
+ }%
+ }%
+}}
+
+\newcommand\foo{$\bullet$}
+\newcommand\foocheck[1]{\tcpeek\z\ifx\foo\z\else
+ \ifx\par\z\else\addcytoks{#1}\fi\fi}
+\tokcycleenvironment\dofoo
+ {\addcytoks{##1}}
+ {\processtoks{##1}}
+ {\ifx\par##1\foocheck{##1}\else\addcytoks{##1}\fi}
+ {\foocheck{##1}}
+
+\newcommand\coleqchek[1]{\tcpeek\z
+ \ifx=\z\tcpop\z\addcytoks{\coloneqq}\else\addcytoks{#1}\fi}
+\tokcycleenvironment\subcoloneqq
+{\tctestifx{:##1}{\coleqchek{##1}}{\addcytoks{##1}}}
+{\processtoks{##1}}
+{\tctestifcon{\if\detokenize{:}\detokenize{##1}}
+ {\coleqchek{##1}}{\addcytoks{##1}}}
+{\addcytoks{##1}}
+
+
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Examples, examples, and more examples}
@@ -292,7 +371,8 @@
\begingroup
-\begin{exampleA}[lefthand width=7.8cm]{}
+\begin{exampleA}[lefthand width=7.8cm]{Apply different directives to
+ characters{,} groups{,} macros{,} and spaces}
\tokcycle{\addcytoks{\underdot{#1}}}
{\addcytoks{\{}\processtoks{#1}
\addcytoks{\}}}
@@ -307,9 +387,13 @@
\subsubsection{Escaping text}
Text between two successive escape characters is bypassed
- by \TokCycle{} and instead echoed to the output register.
-Default escape character is $|$. One can change it with
+ by \TokCycle{} and instead echoed to the output token list
+ (\macname{cytoks}).
+The default escape character is $|$.
+One can change the token-cycle escape character by way of the
\macname{settcEscapechar} macro.
+Below, the character codes are incremented by one; however, the
+ arguments of \macname{rule} are protected against that.
\begingroup
@@ -343,6 +427,7 @@
output token list (\macname{cytoks}).
\medskip
+\begin{minipage}[t]{\textwidth}
Transform characters (+1 ASCII) via unexpandable macro:\medskip
\begin{exampleA}[lefthand width=8.3cm]
@@ -361,6 +446,7 @@
Note how the above detokenization shows the actual transformation
instructions involving \macname{plusl}.
This is done because \macname{plusl} is not expandable.
+\end{minipage}
\par\noindent\hrulefill\medskip
\endgroup
@@ -394,7 +480,6 @@
\altdecytoks
-\medskip
The above detokenization also shows the transformation macro,
\macname{vowelcap}, as part of \macname{cytoks}.
But because \macname{vowelcap} is expandable, we did not need to
@@ -404,9 +489,16 @@
\endgroup
\end{minipage}
+\begin{minipage}[t]{\textwidth}
\begingroup
Capitalize vowels (expanding the character directive)
+\medskip
+By using the \texttt{[x]} option of \macname{addcytoks}, the
+ \macname{vowelcap} is fully expanded prior to being placed in
+ the \macname{cytoks} token list.
+
+
\begin{exampleA}[lefthand width=8.3cm]
{Expanded Character directive}
\tokcycle
@@ -420,11 +512,12 @@
\altdecytoks
-\medskip
Here, we see no vestiges of \macname{vowelcap} in \macname{cytoks},
but only the transformed result.
\endgroup
+\end{minipage}
+\enlargethispage{12pt}
\subsubsection{Unexpanded vs. pre-expanded input stream}
\begingroup
@@ -446,7 +539,6 @@
\altdecytoks
-\medskip
In the above case, vowel-capping is not applied to the date, because
\macname{today} remains unexpanded in the \macname{cytoks} token list.
@@ -468,7 +560,6 @@
\altdecytoks
-\medskip
Here, we see the date subjected to vowel-capping, because the \TokCycle{}
input was expanded prior to the token cycle.
Thus \macname{today} no longer appears \textit{explicitly} in the
@@ -494,7 +585,7 @@
The macro
\macname{stripimplicitgroupingcase} can take three possible
integer arguments: 0 (default) to automatically place unaltered
- implicit group tokens in the output register; 1 to strip
+ implicit group tokens in the \macname{cytoks} output; 1 to strip
implicit group tokens from the output; or $-$1 to instead pass the
implicit group tokens to the Character directive (as implicit
tokens) for separate processing (typically, when detokenization
@@ -547,8 +638,8 @@
involves removing the grouping braces from around the group.
The choice of \macname{processtoks} vs.\@ \macname{addcytoks}
affects whether the tokens inside the group are recommitted
- to \TokCycle{} for processing, or are merely sent to the output
- register in their original unprocessed form.
+ to \TokCycle{} for processing, or are merely sent to the
+ \macname{cytoks} token list in their original unprocessed form.
\medskip
Note that, in these examples, underdots and visible spaces
@@ -700,7 +791,6 @@
\altdecytoks
-\medskip
Without \macname{stripgroupingtrue}, all tasks, A,B,C, and D, would have
been contained within the explicit \texttt{\{...\}} group.
@@ -785,7 +875,7 @@
typesetting of detokenized material.
\medskip It is an unusual application of \TokCycle{} in that it does not
- actually use the \macname{cytoks} token register to collect
+ actually use the \macname{cytoks} token list to collect
its output.
This is only possible because all macros in the input stream
are detokenized, rather than executed.
@@ -821,29 +911,29 @@
\subsubsection{Capitalize all words, including compound and
- parenthetical words}
+ parenthetical words\label{s:cap}}
\begingroup
\begin{exampleC}[]{The \macname{Titlecase} and \texttt{\string\nextcap} macros}
\newcommand\TitleCase[1]{%
- \def\capnext{T}
+ \def\capnext{T}% <- INITIAL ENTRY
\tokcycle
- {\addcytoks{\nextcap{##1}}}
- {\processtoks{##1}}
+ {\nextcap{##1}}
+ {\def\capnext{T}\processtoks{##1}}% <- GROUP ENTRY
{\addcytoks{##1}}
- {\addcytoks{##1\def\capnext{T}}}
+ {\addcytoks{##1}\def\capnext{T}}% <-CAT-10 SPACE
{#1}%
\the\cytoks
}
\newcommand\nextcap[1]{%
\edef\tmp{#1}%
- \tctestifx{-#1}{\def\capnext{T}}{}%
+ \tctestifx{-#1}{\def\capnext{T}}{}% <- TEST FOR HYPHEN
\tctestifcon{\if T\capnext}%
- {\tctestifcon{\ifcat\tmp A}%
- {\uppercase{#1}\def\capnext{F}}%
- {#1}}%
- {#1}%
+ {\tctestifcon{\ifcat\tmp A}% <- TEST FOR NEXT CAT-11
+ {\addcytoks{\uppercase{#1}}\def\capnext{F}}%
+ {\addcytoks{#1}}}%
+ {\addcytoks{#1}}%
}
\end{exampleC}
@@ -859,6 +949,19 @@
(\textit{capitalizing} words).}
\end{exampleA}
+As it is currently set up, a future-capitalization of the next catcode-11 token will be triggered when any of the following is encountered in the input stream:
+\begin{itemize}
+\item The initial entry into the routine
+
+\item a catcode-10 space token
+
+\item a hyphen \texttt{-}
+
+\item upon entry into a group \texttt{\{...\}}
+\end{itemize}
+
+Obviously, additional tests could be added.
+
\endgroup
\subsubsection{Scaling rule dimensions}
@@ -890,7 +993,7 @@
\endgroup
-\subsubsection{String search, including non-regex material}
+\subsubsection{String search, including non-regex material\label{s:searchA}}
\begingroup
@@ -953,8 +1056,8 @@
\begingroup
The problem of counting the significant digits in a number is an interesting
- one, in that leading and trailing zeros are not significant, but zeros
- in the middle of the number are significant.
+ one, in that leading zeros are not significant, trailing zeros are sometimes
+ significant, but zeros in the middle of the number are significant.
In a problem like this, the \macname{cytoks} token list is not used to
carry a transformation of the input tokens, but only to carry a record
of errant tokens from the input (as seen from the helper macro
@@ -977,30 +1080,32 @@
\begin{exampleC}[]{The \macname{sigdigits} macro for counting significant digits}
\newcounter{sigd}
\newcounter{trailingz}
+\newif\iffounddot
+\newif\ifzerosig
\newcommand\sigdigits[1]{%
\setcounter{sigd}{0}% SIGNIFICANT DIGIT COUNT
\setcounter{trailingz}{0}% TRAILING ZERO COUNT
- \def\founddot{F}% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED
- \def\zerosig{F}% MADE T WHEN ZERO BECOMES SIGNIFICANT
+ \founddotfalse% MADE T WHEN DECIMAL HAS BEEN 1ST LOCATED
+ \zerosigfalse% MADE T WHEN ZERO BECOMES SIGNIFICANT
\tokcycle% CYCLE THROOUGH EACH TOKEN
- {\tctestifx{.##1}%
- {\if T\founddot\throwerr\fi\def\founddot{T}}% IF .
- {\tctestifnum{`##1>`/}%
- {\tctestifnum{`##1<`:}%
- {\tctestifnum{##1>0}%
- {\stepcounter{sigd}\def\zerosig{T}\setcounter{trailingz}{0}}% IF 1-9
- {\if T\zerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0
- }%
- {\throwerr}% IF ASCII > `9
- }%
- {\throwerr}% IF ASCII < `0
+ {\tctestifx{.##1}%
+ {\iffounddot\throwerr\fi\founddottrue}% IF .
+ {\tctestifnum{`##1>`/}%
+ {\tctestifnum{`##1<`:}%
+ {\tctestifnum{##1>0}%
+ {\stepcounter{sigd}\zerosigtrue\setcounter{trailingz}{0}}% IF 1-9
+ {\ifzerosig\stepcounter{sigd}\fi\stepcounter{trailingz}}% IF 0
}%
- }% APPLY ABOVE LOGIC FOR CHAR TOKENS
- {\throwerr}% IF BRACES
- {\throwerr}% IF CONTROL SEQUENCE
- {}% IGNORE SPACES
- {#1}% THE ARGUMENT
- \addtocounter{sigd}{-\thetrailingz}% SUBTRACT TRAILING ZEROS
+ {\throwerr}% IF ASCII > `9
+ }%
+ {\throwerr}% IF ASCII < `0
+ }%
+ }% APPLY ABOVE LOGIC FOR CHAR TOKENS
+ {\throwerr}% IF BRACES
+ {\throwerr}% IF CONTROL SEQUENCE
+ {}% IGNORE SPACES
+ {#1}% THE ARGUMENT
+ \iffounddot\else\addtocounter{sigd}{-\thetrailingz}\fi% TRAILING ZEROS
\expandafter\ifx\expandafter\relax\detokenize\expandafter{\the\cytoks}%
\relax\else\setcounter{sigd}{-1}\fi% CHECK FOR INVALID INPUT
\thesigd
@@ -1213,6 +1318,11 @@
As with other \TokCycle{} environments and macros, portions can be
exempted from the count (for example macro arguments) by using the
escape delimiters provided.
+An even more elaborate version of this environment can be found at
+\texttt{https://tex.stackexchange.com/questions/\allowbreak
+ 44618/dynamically-count-and-return-number-of-words-in-a-section/587567\#587567},%
+\hfil in\-cluding a version that runs
+ in Plain \TeX.
\begin{exampleC}[]{The \macname{countem} extended-environment definition
and supporting macros}
@@ -1326,7 +1436,7 @@
}
\begin{exampleB}[text only, righthand width=0.01cm, colback=white]%
-{Effects of various truncation directives}
+{Example: Effects of various truncation directives}
{\footnotesize\ttfamily Input text}: \expandafter\altdetokenize\expandafter
{\thetext}
@@ -1350,10 +1460,339 @@
\endgroup
-\subsection{Advanced topics: implicit tokens and catcode changes}
+\subsection{Look-ahead features}
-\subsubsection{Trap Active Characters (catcode 13)}
+The whole concept of the \TokCycle{} package is to examine the input stream
+ one token at a time and use the directives to process the tokens.
+Occasionally, however, there are tasks that involve several adjacent tokens.
+In the example of section~\ref{s:cap}, for instance, in order to capitalize
+ words, each space token triggers an action on the character that follows.
+To accomplish this without any look-ahead features, the Space directive sets
+ a flag, \verb|\def\capnext{T}|, in order to communicate with a the Character
+ directive about a subsequent token.
+That is a perfectly valid approach to communicate conditional action.
+\medskip
+However, \TokCycle{} also provides look-ahead features, to avoid the necessity
+ of communicating with future tokens in the cycle by way of flag settings.
+These tools allow a directive to directly ``peek'' ahead at the input stream
+ (without disturbing it) and, if desired, push and pop tokens to and from
+ the input stream.
+The package manual should be consulted for a full list of macros provided to
+ facilitate look-ahead capability.
+But here are some examples to look at.
+
+\begingroup
+
+\subsubsection{Count in advance that which follows}
+
+Specify the point values for questions and let tokcycle perform the cumulative
+ tallies in advance, using a single-pass compilation.
+
+\begin{exampleC}[]{The \texttt{pointtracker} environment to count point values
+ in advance}
+\newcounter{pointcount}
+\newcommand\addtopoints[1]{(\textit{#1 point}%
+ \ifnum#1=1\relax\else\textit{s}\fi)}
+\NewEnviron{pointtracker}[2][points]{%
+ \par\bigskip\resettokcycle
+ \setcounter{pointcount}{0}%
+ \Macrodirective{\addcytoks{##1}\tctestifx{\addtopoints##1}{\sumpoints}{}}%
+ \def\tmp{\tokencyclexpress{\large\bfseries#2: \thepointcount\ #1}}%
+ \expandafter\tmp\BODY\endtokencyclexpress
+}
+\def\sumpoints{\tcpop\Q\addtocounter{pointcount}{\Q}\tcpushgroup{\Q}}
+\end{exampleC}
+
+\begin{exampleA}[lefthand width=6.2cm]{Demo of the pointtracker environment}
+\begin{pointtracker}[total points]{\Large Science Test}
+\begin{pointtracker}{Advanced Geology}
+ \begin{itemize}
+ \item Count the fingers on your left hand. Are there more than 9?
+ \addtopoints{1}
+ \item Draw a rectangle with five edges. \addtopoints{3}
+ \end{itemize}
+\end{pointtracker}
+\begin{pointtracker}{Meteorology}
+ \begin{itemize}
+ \item How many borgs does it take to brew a Danish beer?
+ \addtopoints{2}
+ \item What is the meaning of life? \addtopoints{4}
+ \end{itemize}
+\end{pointtracker}
+\end{pointtracker}
+\end{exampleA}
+
+Note that this example has several neat features.
+The points for Geology, Meteorology, and the Test total are not specified,
+ but calculated.
+They are calculated based on content that appears later in the environment.
+Traditionally, multiple compilations would be required with external data
+ written to the aux file, but that does not occur here---there is only
+ a single compilation.
+Finally, becaue we are nesting the \TokCycle{} part of \texttt{pointtracker}
+ inside of a \macname{NewEnviron}, we see that the \texttt{pointtracker}
+ environment can be nested, without adverse effect.
+
+\endgroup
+
+\subsubsection{Modify the arguments of every instance of a particular macro}
+
+\begingroup
+
+This example, through the use of the \macname{scalerule} macro, shows how
+ the \macname{rule} arguments, including the optional argument, can be
+ popped off the input stream, adjusted, and placed back in the stream.
+
+\begin{exampleC}[]{The \macname{scalerule} macro to scale all
+ \macname{rule} arguments}
+\newcommand\scalerule[1]{%
+ \tcpeek\zA
+ \ifx[\zA\tcpopuntil]\opt\fi% IF PRESENT, POP OPTIONAL ARG
+ \tcpop\ArgA% POP ARG1
+ \tcpop\ArgB% POP ARG2
+ \tcpushgroup[x]{#1\dimexpr\ArgB}% PUSH BACK IN REVERSE (LIFO) FASHION
+ \tcpushgroup[x]{#1\dimexpr\ArgA}
+ \ifx[\zA\tcpush[x]{\expandafter\scaleopt\opt{#1}}\fi
+}
+\def\scaleopt[#1]#2{[#2\dimexpr#1]}
+\end{exampleC}
+
+\begin{exampleA}[lefthand width=9.2cm]{Scale every \macname{rule} argument}
+\resettokcycle
+\settcEscapechar{!}% SO AS NOT TO INTERFERE WITH `|' TOKS IN tabular
+\Macrodirective{\addcytoks{#1}\ifx\rule#1\scalerule{2}\fi}
+\tokencyclexpress
+The rules appearing in the following input stream are literally
+!\rule[-2pt]{1.5ex}{1.5ex}! and !\rule{2ex}{3pt}!:
+
+My environment can include all kinds of rules
+ \rule[-2pt]{1.5ex}{1.5ex} and other macros, such as \today{} and
+\begin{tabular}{c|c}
+23 & 454\\\hline
+x & y
+\end{tabular}
+but each rule \rule{2ex}{3pt} is grown by a factor of two.
+\endtokencyclexpress
+\end{exampleA}
+\settcEscapechar{|}
+
+\endgroup
+
+
+\subsubsection{Protect arguments of specified macro from Character
+ directive modification}
+
+\begingroup
+Tokcycle provides the escape character $|$ to delimit tokens to be protected
+ from processing by tokcycle directives.
+But that requires the input stream to be adjusted to include these delimiters.
+Here's an alternative, not requiring input stream adjustment.
+The Macro directive protects all \macname{rule} arguments against contamination
+ by the Character directive.
+
+\medskip
+When a \macname{rule} is encountered, \macname{protectruleargs} is called
+ to pop the rule arguments from the input stream and place them directly
+ in the \macname{cytoks} token list, out of reach from the Character directive.
+
+
+\begin{exampleC}[]{The \macname{protectruleargs} macro to prevent Character directive
+ contamination of arguments}
+\newcommand\protectruleargs[1]{\ifx\rule#1%
+ \tcpeek\zA
+ \ifx[\zA\tcpopuntil]\opt
+ \addcytoks[1]{\opt}\fi% IF OPTIONAL ARG PRESENT, POP AND SAVE TO \cytoks
+ \tcpopliteral\ArgA% MUST POP ARG 1 WITH BRACES
+ \addcytoks[1]{\ArgA}%
+ \tcpopliteral\ArgB% MUST POP ARG 2 WITH BRACES
+ \addcytoks[1]{\ArgB}%
+ \fi
+}
+\end{exampleC}
+
+Here, every character is typeset with a slash suffix.
+If this action were to be imposed to \macname{rule} argument characters,
+ it would destroy their integrity.
+Fortunately, when the Macro directive detects a \macname{rule}, it can
+ take evasive action.
+
+\begin{exampleA}[lefthand width=9.1cm]{Protect every \macname{rule} argument
+ from Character directive contamination}
+\resettokcycle
+\Characterdirective{\addcytoks{#1\textup{/}\allowbreak}}
+\Macrodirective{\addcytoks{#1}\protectruleargs{#1}}
+\tokencyclexpress
+My \textit{environment} modifies each character token, but rule
+ \rule[-4pt]{1.5ex}{1.5ex} arguments \rule{2ex}{3pt} are protected.
+\endtokencyclexpress
+\end{exampleA}
+\endgroup
+
+
+\begingroup
+
+\subsubsection{Find a string of character tokens}
+
+Earlier in this manuscript, in section~\ref{s:searchA}, a token cycle
+ was developed to search a token stream for a string match.
+Here, we do something similar, except we use the new ``look ahead'' features
+ of \TokCycle{} to achieve it.
+In this case, however, the search string cannot cross space or group
+ boundaries.
+The \macname{matchnextchar} macro demonstrates a use of \macname{tcpopappto},
+ which is one of the newly introduced \TokCycle{} commands; instead of
+ popping an argument from the input stream \textit{into} a macro, it pops it
+ and \textit{appends} it to a macro.
+
+\begin{exampleC}[]{The \macname{findstring} macro to perform a look{-}ahead string
+ search}
+\newcommand\findstring[4]{% {CURRENT TOK}{MATCH STRING}{SUCCESS OP}{FAIL CMDS}
+ \foundstringfalse% DEFAULT CONDITION
+ \def\poptoks{}% INITIALIZATION. IF MATCH FAILS, MUST TO RESTORE POPPED TOKENS
+ \expandafter\matchnextchar\ifx#1#2\relax\relax\fi\empty\relax
+ \iffoundstring\addcytoks{#3{#2}}\else\tcpush{\poptoks}#4\fi
+}
+\def\matchnextchar#1#2\relax{\tctestifx{#1\empty}{}{% NO MATCH FROM 1ST TOKEN
+ \tctestifx{\relax#1}{%
+ \foundstringtrue% NOTHING LEFT TO COMPARE; MATCH FOUND
+ }{%
+ \tcpeek\Q% WITHOUT POPPING...
+ \tctestifx{\Q\bgroup}{}{% PEEK FOR GROUPS OR...
+ \tctestifx{\Q\tcsptoken}{}{% SPACES THAT INDICATE MATCH FAILURE
+ \tcpopappto\poptoks% POP TOKEN AND APPEND TO \poptoks
+ \expandafter\tctestifx\expandafter{\Q#1}{%
+ \matchnextchar#2\relax% STILL MATCHING; KEEP GOING
+ }{}%
+ }%
+ }%
+ }%
+}}
+\end{exampleC}
+
+\begin{exampleA}[lefthand width=8cm]{Search for the string ``red''}
+\resettokcycle
+\Characterdirective{\findstring{#1}{red}%
+ {\textcolor{purple}}{\addcytoks{#1}}}
+\tokencyclexpress
+Can I \textit{find red in the text}, even in the
+ MIDredDLE of words and groups?
+\endtokencyclexpress
+\end{exampleA}
+
+\endgroup
+
+
+\subsubsection{Eliminate \textit{prior} spaces and pars before a particular
+ macro}
+
+The goal here is to eliminate spaces and paragraphs that occur
+ \textit{prior} to the occurrence of the macro \macname{foo}, which
+ sets a \macname{bullet}.
+Looking backwards is not part of the \TeX{} approach.
+Nonetheless, the problem can, with a token cycle, be turned on its head,
+ such that all occurrences of spaces and \macname{par} can be made
+ to look for a future instance of \macname{foo}.
+
+\medskip
+Here, if a space or \macname{par} is detected, \macname{foocheck} is called.
+ It peeks at the token that follows the space or \macname{par}.
+If the peeked argument is a \macname{par} or \macname{foo} itself, it discards
+ the current space or \macname{par}.
+
+\begingroup
+
+\begin{exampleC}[]{The \macname{dofoo} environment to remove prior spaces and
+ paragraphs before \macname{foo}}
+\newcommand\foo{$\bullet$}
+\newcommand\foocheck[1]{\tcpeek\z\ifx\foo\z\else
+ \ifx\par\z\else\addcytoks{#1}\fi\fi}
+\tokcycleenvironment\dofoo
+ {\addcytoks{##1}}
+ {\processtoks{##1}}
+ {\ifx\par##1\foocheck{##1}\else\addcytoks{##1}\fi}
+ {\foocheck{##1}}
+\end{exampleC}
+
+\begin{exampleA}[lefthand width=7cm]{Remove spaces and paragraphs before
+ \macname{foo}}
+\dofoo
+Lorem ipsum dolor sit amet. (No foo)
+
+Lorem ipsum dolor sit amet. \foo
+
+Lorem ipsum dolor sit amet.\foo
+
+Lorem ipsum dolor sit amet.
+\foo
+
+Lorem ipsum dolor sit amet.
+
+\foo
+\enddofoo
+\end{exampleA}
+\endgroup
+
+
+\subsubsection{Replace := with \macname{coloneqq} (\texttt{mathtools}
+ package required)}
+
+In most cases, creating a macro to signify a composite symbol makes the most
+ sense.
+However, sometimes you don't realize that until you've typed in a lot of code
+ that fails to use such a macro.
+In such a situation, \TokCycle{} can help.
+
+\medskip
+In the \texttt{mathtools} package, there is a symbol, \macname{coloneqq},
+ a colon followed by an equal sign, in which the colon has been
+ vertically adjusted to be symmetric with respect to the \texttt{=} sign.
+The adjustment is slight, but noticeable.
+A token cycle can be set up to look for instances of the \texttt{:} character,
+ peek ahead to see if it is followed by an \texttt{=} character and, if so,
+ replace the pair with \macname{coloneqq}.
+However, the process is further complicated, if one is a user of
+\macname{usepackage[french]\{babel\}}, where the \texttt{:} character is is made
+ an active character.
+
+\medskip
+Therefore, in this case, not only does \texttt{:} need to be located by
+ the Character directive (in the normal case), but it also need be searched
+ for by the Macro directive, in the event that
+ \macname{usepackage[french]\{babel\}} is being employed.
+In the example below, there are two such colons that satisfy the criterion for
+ replacement.
+
+\begingroup
+
+\begin{exampleC}[]{The \macname{coleqchek} macro and the \macname{subcoloneqq}
+ environment}
+\newcommand\coleqchek[1]{\tcpeek\z
+ \ifx=\z\tcpop\z\addcytoks{\coloneqq}\else\addcytoks{#1}\fi}
+\tokcycleenvironment\subcoloneqq
+{\tctestifx{:##1}{\coleqchek{##1}}{\addcytoks{##1}}}% <- IF : A CHARACTER
+{\processtoks{##1}}
+{\tctestifcon{\if\detokenize{:}\detokenize{##1}}% <- IF : AN ACTIVE MACRO
+ {\coleqchek{##1}}{\addcytoks{##1}}}
+{\addcytoks{##1}}
+\end{exampleC}
+
+\begin{exampleA}[lefthand width=6cm]{replace {:=} with \macname{coloneqq}}
+\subcoloneqq
+\LARGE Compare: $::=$
+
+$a := b$ vs
+ $a \ordinarycolon= b$ vs
+ $a:b$.
+\endsubcoloneqq
+\end{exampleA}
+
+\endgroup
+
+\section{Advanced topics: implicit tokens and catcode changes}
+
+\subsection{Trap active characters (catcode 13)}
+
%\begin{minipage}[t]{\textwidth}
\begingroup
Active characters in the \TokCycle{} input stream are processed in
@@ -1408,10 +1847,12 @@
are made before reaching \TokCycle{} processing.
\svt hey are, thus, no longer detected as active, unless
\macname{noexpand} is applied before the pre-expansion.
-In this example, the `\texttt{!}' that is not \macname{noexpand}\kern.3pted
- is converted to a `\texttt{?}' prior to reaching \TokCycle{}
- processing (and thus, not detected as \macname{active}).
-Likewise, the `\texttt{T}' is pre-converted to a `\texttt{+}':
+In this example, the `\texttt{!}' that is \textit{not} subject to
+ a \macname{noexpand} is converted to a `\texttt{?}' prior to
+ reaching \TokCycle{} processing (and thus, not detected as
+ \macname{active}).
+However, the `\texttt{T}' is not pre-converted to a `\texttt{+}',
+ because it is implicit, and not a definition:
\catcode`!=\active
\def !{?}
@@ -1424,7 +1865,13 @@
\detokenize\expandafter{\the\cytoks}
\end{exampleA}
-However, pre-tokenization does not suffer this behavior:
+\begingroup
+\catcode`T=11
+\catcode`!=12
+However, pre-tokenization does not suffer this behavior.
+The first `\texttt{!}' is tokenized as active during
+ its absorption into \macname{tmp}, as shown in the next example:
+\endgroup
\begin{exampleA}[lefthand width=7.8cm]{Pre-tokenized input stream
does not affect active characters}
\def\tmp{This is a test!!}
@@ -1473,7 +1920,7 @@
\end{exampleA}
\endgroup
-\subsubsection{Trap Catcode 6 (explicit \& implicit) tokens}
+\subsection{Trap catcode 6 (explicit \& implicit) tokens}
\begingroup
@@ -1569,7 +2016,7 @@
\endgroup
-\subsubsection{Trap implicit tokens in general}
+\subsection{Trap implicit tokens in general}
\begingroup
@@ -1656,7 +2103,7 @@
in the input stream as duplicate, e.g. \texttt{\#\#}.
-\subsubsection{Changing grouping tokens (catcodes 1,2)}
+\subsection{Changing grouping tokens (catcodes 1,2)}
\begingroup
@@ -1755,7 +2202,7 @@
\endgroup
-\subsubsection{Catcode 10 space tokens\label{s:spc}}
+\subsection{Catcode 10 space tokens\label{s:spc}}
\begingroup
@@ -1771,17 +2218,19 @@
\ddbend
-Also, if the implicit space is an active character, \macname{ifactivetok}
- is also set, for the user's benefit.
-Likewise, \macname{ifactivechar} is also checked for active spaces, to see
+If the implicit space is \textit{also} an active character,
+ \macname{ifactivetok} is also set, for the user's benefit.
+Likewise, \macname{ifactivechar} is also checked, to see
if the charcode of the active space is still, indeed, active.
While \verb|#1| may be used to represent the space in the space directive,
- a special technique is required to recover the detokenized name of the
+ a special technique may be required to recover the detokenized name of the
active space character.
-In particular, \verb|#1| will, for active spaces, contain a generic
- implicit space token, \macname{tc at sptoken}.
-However, the name of the associated active character in the input stream
- will be defined as a cat-12 token in \macname{theactivespace}.
+In particular, if the active token is an implicit space, but the charcode
+ of that token is no longer active, \verb|#1| will, for this case,
+ contain a generic implicit space token, \macname{tcsptoken}.
+However, for all active space tokens, the name of the associated active
+ character in the input stream will be defined as a cat-12 token in
+ \macname{theactivespace}.
\medskip
Note in the following examples that cat-10 tokens do \textit{not}
@@ -1827,16 +2276,10 @@
\end{exampleA}
The next two examples introduce implicit and active-implicit spaces.
-The latter example also demonstrates detokenization of such spaces.
+In the first example, functionally identical to the prior example,
+ we see that \TokCycle{} can digest and direct such
+ tokens to the Space directive, here in the form of \macname{z}.
-\medskip
-Here is a useful expandable macro for detokenizing space tokens
- in the context of \TokCycle{}.
-It can process not only explicit cat-10 spaces, but also implicit and
- active-implicit spaces.
-To add it to the \macname{cytoks} token list, make sure you use the
- \verb|[x]| fully expanded option to \macname{addcytoks}.
-
%%%
\resettokcycle
\Characterdirective{\addcytoks{\underdot{#1}}}
@@ -1853,40 +2296,63 @@
\endtokencyclexpress
\end{exampleA}
-\begin{minipage}[t]{\textwidth}
-\begin{exampleC}[]{The \macname{detokenizespacetok} macro for handling
- implicit and active-implicit spaces}
-\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}%
- {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}}
-\end{exampleC}
+In the second example, an active implicit space token is created, \texttt{Q},
+ and stored in a definition, \macname{tmp}.
+A \macname{string} is applied by the token cycle at one point to the
+ space tokens and we see that the proper implicit tokens are decoded, whether
+ they are active or macro.
-\begin{exampleA}[lefthand width=10cm]{Active Implicit spaces work{,} too}
+\medskip
+The one case deserving special note is the final case of the example,
+ in which the catcode of \texttt{Q}, the active character, is restored
+ to a value of 12, before the previously tokenized, active-implicit
+ \texttt{Q} is passed to the token cycle.
+In such a case, where future incantations of \texttt{Q} are no longer active,
+ then the active-implicit space token that gets passed to the
+ Space directive (by way of \macname{tmp}) is taken as a generic
+ active-implicit \macname{tcsptoken}, instead of the original
+ active-implicit token (\texttt{Q}).
+
+\medskip
+Nonetheless, the original string of the active-implicit token (that is, a
+ cat-12 \texttt{Q}) is preserved in \macname{theactivespace}.
+The reason for this behavior is that the absorption of spaces in \TokCycle{}
+ is a destructive process, in which the spaces are absorbed as strings,
+ so that a proper interpretation of their implicitness can be ascertained.
+With the character no longer active, it would be more difficult to
+ recreate an active-implicit version of \textit{that} token to
+ pass to the Space directive.
+Thus, a generic active-implicit space token is, instead, employed.
+
+
+\begin{exampleA}[lefthand width=9.05cm]{Active Implicit spaces work{,} too}
\resettokcycle
-\Characterdirective{\addcytoks{\underdot{#1}}}
\def\:{\let\z= } \: %
\catcode`Q=\active %
\def\:{\let Q= } \: %
\catcode`\_=10 %
-\tokencyclexpress
-xQx x_x\z{}x%
-\endtokencyclexpress
+\def\tmp{xQx x_x\z{}x}
+\expandafter\tokencyclexpress\tmp\endtokencyclexpress
-\resettokcycle
\Spacedirective{\addcytoks{\textcolor{cyan}\bgroup}%
- \addcytoks[x]{(\detokenizespacetok{#1})}%
+ \addcytoks[x]{(\string#1\ifactivetok
+ \ifimplicittok\ifactivechar\else\noexpand
+ \textcolor{red}{:\theactivespace}\fi\fi\fi)}%
\addcytoks{\egroup}}
\medskip
-\tokencyclexpress
-xQx x_x\z{}x%
-\endtokencyclexpress
+\expandafter\tokencyclexpress\tmp\endtokencyclexpress
+
+\medskip
+\catcode`Q=12
+\expandafter\tokencyclexpress\tmp\endtokencyclexpress
+
\end{exampleA}
-\end{minipage}
\endgroup
-\subsubsection{Changes to catcode 0}
+\subsection{Changes to catcode 0}
\begingroup
Modified: trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex
===================================================================
--- trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex 2021-05-28 20:48:24 UTC (rev 59376)
+++ trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex 2021-05-28 20:48:44 UTC (rev 59377)
@@ -1,7 +1,7 @@
\def\tcname {tokcycle}
-\def\tcver {1.3}
+\def\tcver {1.4}
%
-\def\tcdate {2021/03/10}
+\def\tcdate {2021/05/27}
%
% Author : Steven B Segletes, Christian Tellechea (contributor)
% Maintainer : Steven B Segletes
@@ -52,16 +52,18 @@
\long\def\tc at exsecond#1#2{#2}
\long\def\tc at testxifx{\tc at earg\tctestifx}
\long\def\test at ifmacro#1{\tctestifcatnx#1\relax}
-\def\tc at addtoks#1#2{\toks#1\expandafter{\the\toks#1#2}}
+\def\tc at addtoks#1#2{\toks#1\expandafter{\the\toks#1 #2}}
\def\add at tcdepth{\advance\tcdepth 1\relax}
-\def\sub at tcdepth{\advance\tcdepth-1\relax}\def\tc at resetifs{\activetokfalse\implicittokfalse\tc at implicitgrpfalse
- \catSIXfalse\activecharfalse\activetokunexpandablefalse
- \def\theactivespace{}}
+\def\sub at tcdepth{\advance\tcdepth-1\relax}
+\def\tc at resetifs{\activetokfalse\implicittokfalse\tc at implicitgrpfalse\catSIXfalse
+ \activecharfalse\activetokunexpandablefalse\def\theactivespace{}}
\long\def\count at stringtoks#1{\tc at earg\count at toks{\string#1}}
\long\def\count at toks#1{\the\numexpr-1\count@@toks#1.\tc at endcnt}
\long\def\count@@toks#1#2\tc at endcnt{+1\tc at ifempty{#2}{\relax}{\count@@toks#2\tc at endcnt}}
\def\sv at hash{##}
+\def\def at popname{\expandafter\def\tc at popname}
% EXTERNAL TOOLS
+\let\tcsptoken= \tc at sptoken
\long\def\tctestifcon#1{#1\expandafter\tc at exfirst\else\expandafter\tc at exsecond\fi}
\long\def\tctestifcatnx#1#2{\tctestifcon{\ifcat\noexpand#1\noexpand#2}}
\long\def\tctestifx#1{\tctestifcon{\ifx#1}}
@@ -73,6 +75,7 @@
\newif\ifactivetok
\newif\ifactivechar
\newif\ifactivetokunexpandable
+\newif\ifspacepopped
\newtoks\cytoks
\long\def\tokcycleenvironment#1#2#3#4#5{\expandafter\def\expandafter#1%
\expandafter{\expandafter\let\csname end\expandafter\tc at gobble
@@ -122,7 +125,9 @@
{\implicittoktrue\tctestifcon{\if\expandafter\@firstoftwo\string\\\tc at str}%
{\expandafter\@@@spcT\tc at gobble}% IMPLICIT MACRO SPACE
{\activetoktrue\@@@@spcT}}}% IMPLICIT ACTIVE SPACE
-\def\@@@@spcT#1{\def\theactivespace{#1}\trapactivechar{#1}\@@spcT{\tc at sptoken}}
+\def\@@@@spcT#1{\def\theactivespace{#1}\trapactivechar{#1}\ifactivechar\edef\tc at spc
+ {\scantokens{#1\noexpand}}\else\def\tc at spc{\tcsptoken}\fi\expandafter\@@spcT
+ \expandafter{\tc at spc}}
\def\@@@spcT{\csmk{\expandafter\@@spcT\thecs}}
\def\@@spcT#1{\@spcT{#1}\@tokcycle}
% GROUP DECODE
@@ -161,7 +166,7 @@
\addcytoks\expandafter{\expandafter#1}}}
\def\settcGrouping#1{\def\@tmp##1{#1}\tc at defx\@@tmp{\@tmp{\the\cytoks}}%
\tc at earg\defineexit at grouped{\@@tmp}}
-% FAUX TOKENIZATION OF COMMAND NAME (WHEN cat0 TOKEN HAS BEEN MADE cat12)
+% FAUX TOKENIZATION OF COMMAND NAME (WHEN \ AND COMMAND-NAME TOKS ARE NOW cat12)
\def\csmk#1{\def\csaftermk{#1}\toks0{}\@csmkA}
\def\@csmkA{\futurelet\@tmp\@csmkB}
\def\@csmkB{\tctestifx{\@tmp\tc at sptoken}%
@@ -184,6 +189,38 @@
%% WILL ALSO TRAP ACTIVE \let TO PRIMITIVES AS IMPLICIT; UNDO LATER IN \can at absorb@@
\def\trapactivetokunexpandable#1{\tctestifcon{\expandafter\if
\detokenize{#1}#1}{}{\activetoktrue\activetokunexpandabletrue\implicittoktrue}}
+% FEATURES TO LOOK-AHEAD INTO THE INPUT STREAM (INTRODUCED v1.4)
+\long\def\tcpeek#1#2\@tokcycle{\def\tc at tmp{\ifx#1\endtokcycraw
+ \let#1=\empty\fi#2\@tokcycle}\futurelet#1\tc at tmp}%___________________________PEEK_
+\def\tcpopliteral#1{\tcpopwhitespace#1\tcpeek\@tmp\ifx\@tmp\bgroup
+ \tcpop\@tmp\def\tc at tmp{\tcappto#1from}\expandafter\tc at tmp\expandafter
+ {\expandafter{\@tmp}}\else\tcpopappto#1\fi}
+\def\tcpop{\long\def\tc@@@pop##1{\tctestifx{\endtokcycraw##1}{\def at popname{}%
+ \tc at tmp\endtokcycraw}{\def at popname{##1}\tc at tmp}}\tc at pop}
+\long\def\tcpopuntil#1{\long\def\tc@@@pop##1#1{\def at popname{##1#1}\tc at tmp}\tc at pop}
+\long\def\tc at pop#1#2\@tokcycle{\def\tc at popname{#1}\def\tc at tmp
+ {#2\@tokcycle}\futurelet\tc at futuretok\tc@@pop}
+\def\tc@@pop{\tc at trapescape\tc at futuretok{\def at popname{}\tc at tmp}{\tctestifx{%
+ \endtokcycraw\tc at futuretok}{\def at popname{}\tc at tmp}{\tctestifx{\tc at sptoken%
+ \tc at futuretok}{\spacepoppedtrue\tc@@@pop}{\spacepoppedfalse\tc@@@pop}}}}%_____POP_
+\def\tcappto#1from#2{%
+ \expandafter\tc at defx\expandafter#1\expandafter{\expandafter#1#2}}
+\def\tcpopliteralappto#1{\tcpopliteral\@tmp\tcappto#1from\@tmp}
+\def\tcpopappto#1{\tcpop\@tmp\tcappto#1from\@tmp}%___________________________APPEND_
+\long\def\tcpopwhitespace#1#2\@tokcycle{\def\tc at popname{#1}\def at popname{}%
+ \def\tc at tmp{#2\@tokcycle}\futurelet\tc at futuretok\tc at popspc}
+\def\tc at popspc{\tctestifx{\tc at sptoken\tc at futuretok}{\discern at space}{\tc at tmp}}
+\def\discern at space{\begingroup\def\@@spcT##1{\tctestifcon\ifimplicittok{\gdef
+ \tc at nxt{\tc at tmp##1}}{\gdef\tc at nxt{\def at popname{##1}\tc at tmp}}\endgroup
+ \tc at nxt}\stringify\@@@@@@spcT}%_____________________________________________SPACE_
+\long\def\@tcpush#1#2\@tokcycle{\def\tc at tmp{#2\@tokcycle}\expandafter\tc at tmp#1}
+\long\def\@tcpushgroup#1#2\@tokcycle{\def\tc at tmp{#2\@tokcycle}\expandafter\tc at tmp
+ \expandafter{#1}}
+% ...BORROW \addcytoks OPTIONAL ARGUMENT EXPANSION FEATURE FOR \tcpush[group]
+\def\tcpush{\bgroup\long\def\tc at addtotoks##1{\egroup
+ \@tcpush{##1}}\futurelet\nxttok\addcytoks at A}
+\def\tcpushgroup{\bgroup\long\def\tc at addtotoks##1{\egroup
+ \@tcpushgroup{##1}}\futurelet\nxttok\addcytoks at A}%___________________________PUSH_
% EXPRESS-INTERFACE - ALLOWS TO EXTERNALLY DEFINE DIRECTIVES
\def\Characterdirective{\def\@chrT##1}
\def\Groupdirective{\long\def\@grpT##1}
@@ -247,7 +284,7 @@
in the definition \theactivespace. The flags \implicittok and \activetok
are both set true, and the \activechar flag is checked, as well.
-v1.3 2021/03/09
+v1.3 2021/03/10
- Introduced \xtokcycleenvironment, similar to \tokcycleenvironment, but
allows two additional arguments, defining the setup and trailing code
that will be run prior to the invocation and following conclusion of the
@@ -283,3 +320,16 @@
\tcdepth associated with the Group directive (also, outer token cycle
operates with depth = 0 rather than 1).
- Renamed \@defgroupedcytoks to \defineexit at grouped, to better match function.
+
+v1.4 2021/05/26
+ - Concerning the \@@@@spcT macro: previously, active-implicit spaces
+ were passed to the Spacedirective as \tc at sptoken, with the \string of the
+ active char passed in \theactivespace. Now, the active-implicit space token
+ itself is passed to the Spacedirective instead of \tc at sptoken, but only *IF*
+ the charcode of that character is currently active; otherwise, a generic
+ implicit space, \tcsptoken is passed.
+ - Introduced a set of "look ahead" macros: \tcpeek, \tcpop, \tcpopliteral,
+ \tcpopappto, \tcpopliteralappto, \tcappto#1from#2, \tcpopuntil,
+ \tcpopwhitespace, \tcpush, and \tcpushgroup.
+ - Bug fixed in \tc at addtoks definition, in the event that #2 was a number.
+
More information about the tex-live-commits
mailing list.