texlive[56501] Master/texmf-dist: tokcycle (1oct20)
commits+karl at tug.org
commits+karl at tug.org
Thu Oct 1 23:05:20 CEST 2020
Revision: 56501
http://tug.org/svn/texlive?view=revision&revision=56501
Author: karl
Date: 2020-10-01 23:05:19 +0200 (Thu, 01 Oct 2020)
Log Message:
-----------
tokcycle (1oct20)
Modified Paths:
--------------
trunk/Master/texmf-dist/doc/generic/tokcycle/README
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/README
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/README 2020-10-01 20:42:55 UTC (rev 56500)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/README 2020-10-01 21:05:19 UTC (rev 56501)
@@ -3,12 +3,12 @@
Author: Steven B Segletes, Christian Tellechea (contributor)
-Files : 1) tokcycle.tex
- 2) tokcycle.sty
+Files : 1) tokcycle.tex (tex/generic/tokcycle)
+ 2) tokcycle.sty (tex/latex/tokcycle)
3) tokcycle-doc.tex
- 4) tokcycle-doc.pdf
- 5) tokcycle-examples.tex
- 6) tokcycle-examples.pdf
- 7) README
+ 4) tokcycle-doc.pdf (doc/latex/tokcycle)
+ 5) tokcycle-examples.tex (doc/latex/tokcycle)
+ 6) tokcycle-examples.pdf (doc/latex/tokcycle)
+ 7) README (doc/latex/tokcycle)
Released under the LaTeX Project Public License v1.3c or later, see http://www.latex-project.org/lppl.txt
\ No newline at end of file
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.pdf
===================================================================
(Binary files differ)
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex 2020-10-01 20:42:55 UTC (rev 56500)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-doc.tex 2020-10-01 21:05:19 UTC (rev 56501)
@@ -80,6 +80,7 @@
The package provides a number of options
for handling groups.
\end{quote}
+\enlargethispage{1.4\baselineskip}
\tableofcontents
@@ -299,8 +300,8 @@
The \texttt{[x]} option employs the \macname{expanded} primitive to maximally
expand the argument before appending it to \macname{cytoks}.
-The \texttt{[x]} option will prove useful when the \Character{} directive involves
- a transformation that is fully expandable.
+The \texttt{[x]} option will prove useful when the \Character{} or other
+ directives involve a transformation that is fully expandable.
Its use will allow the expanded
result of the transformation to be placed in the token register, rather
than the unexpanded transformation instructions.
@@ -464,6 +465,35 @@
ii)~implicit space tokens (e.g., \macname{@sptoken}) will be processed
by the \Space{} directive.
+\subsubsection{Active-Implicit Tokens, Including Spaces}
+
+One may occasionally run across a token that is both active and implicit.
+For example, in the following code, \texttt{Q} is made both active and implicit:
+
+\begin{verbbox}[\vbdelim]
+\catcode`Q=\active
+\let Qx
+\end{verbbox}
+\medskip\theverbbox\medskip\noindent
+
+\noindent
+In general, both \macname{ifactivetok} and \macname{ifimplicittok}
+ tests can be performed together to determine such cases.
+
+This is true even in the case of active-implicit catcode-10 spaces, which
+ are always processed through the Space directive.
+However, because of the process \TokCycle{} uses in digesting space
+ tokens, the actual token passed to the Space directive (so-called
+ \verb|#1|), when an active-implicit space is encountered in the input
+ stream, is a generic implicit space token named \macname{tc at sptoken}.
+However, the catcode-12 version of the active character that
+ produced it will be, for that moment, retained in a definition
+ named \macname{theactivespace}.
+This can be useful if detokenization is required of the spaces.
+Such an example is described in the \texttt{tokcycle-examples}
+ adjunct document.
+
+
\subsubsection{Parameter (cat-6) tokens (e.g., \texttt{\#}): \macname{ifcatSIX}}
Typically, category-code 6 tokens (like \texttt{\#}) are used to
@@ -589,7 +619,7 @@
Surely, there are more cases associated with arcane catcode-changing
syntax that are not accounted for;
I encourage you to bring them to my attention.
-If I can't fix them, I can at least disclaim and declaim them.
+If I can't fix them, I can at least disclaim and declaim them below:
\begin{itemize}
@@ -633,7 +663,7 @@
\section*{Source Code}
\Large\texttt{tokcycle.sty}
-\verbfilenobox[\footnotesize]{tokcycle.sty}\vspace{2in}
+\verbfilenobox[\footnotesize]{tokcycle.sty}\vspace{0.5in}
\noindent\texttt{tokcycle.tex}
\verbfilenobox[\footnotesize]{tokcycle.tex}
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.pdf
===================================================================
(Binary files differ)
Modified: trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex
===================================================================
--- trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex 2020-10-01 20:42:55 UTC (rev 56500)
+++ trunk/Master/texmf-dist/doc/generic/tokcycle/tokcycle-examples.tex 2020-10-01 21:05:19 UTC (rev 56501)
@@ -194,6 +194,8 @@
\newcommand\nextcmptok[1]{\rotoradd{\rotcytoks{#1}}{\addcytoks{#1}}\testmatch{\tmp}}
\newcommand\nextctltok[1]{\stepcounter{runcount}\addcytoks{#1}}
+\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}%
+ {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Examples, examples, and more examples}
@@ -873,6 +875,8 @@
directive;
If \macname{let} to a control sequence or defined via \macname{def},
they may be identified in the Macro directive.
+For information on how to process active spaces, please refer to
+ section~\ref{s:spc}.
\let\svt T
\Characterdirective{\tctestifcon\ifactivetok
{\addcytoks{\fbox{#1-chr}}}{\addcytoks{#1}}}
@@ -1257,12 +1261,13 @@
\endgroup
-\subsubsection{Catcode 10 space tokens}
+\subsubsection{Catcode 10 space tokens\label{s:spc}}
\begingroup
Here we demonstrate that \TokCycle{} can handle arbitrary redesignation
- of tokens to cat-10, as well as implicit space tokens.
+ of tokens to cat-10, as well as implicit space tokens (both implicit
+ macro spaces and active-implicit character spaces).
\ddbend While it should seem natural, we note that implicit
space tokens are directed to the Space directive rather than
@@ -1269,8 +1274,22 @@
the Character directive.
However, \macname{ifimplicittok} may still be used to
differentiate an explicit space from an implicit one.
+
+
+\ddbend
+Also, if the implicit space is an active character, \macname{ifactivetok}
+ is also set, for the user's benefit.
+Likewise, \macname{ifactivechar} is also checked for active spaces, to see
+ if the charcode of the active space is still, indeed, active.
+While \verb|#1| may be used to represent the space in the space directive,
+ a special technique is required to recover the detokenized name of the
+ active space character.
+In particular, \verb|#1| will, for active spaces, contain a generic
+ implicit space token, \macname{tc at sptoken}.
+However, the name of the associated active character in the input stream
+ will be defined as a cat-12 token in \macname{theactivespace}.
+
\medskip
-
Note in the following examples that cat-10 tokens do \textit{not}
get under-dots.
The next three examples all use the same input, but with different
@@ -1313,22 +1332,64 @@
\fbox{a_c d} b_g\itshape f\upshape\endtokencycle
\end{exampleA}
+The next two examples introduce implicit and active-implicit spaces.
+The latter example also demonstrates detokenization of such spaces.
+
+\medskip
+Here is a useful macro for detokenizing space tokens in the context of
+ \TokCycle{}.
+It can process not only explicit cat-10 spaces, but also implicit and
+ active-implicit spaces.
+To add it to the \macname{cytoks} token list, make sure you use the
+ \verb|[x]| fully expanded option to \macname{addcytoks}.
+
+\medskip
+\begin{exampleC}[]{The \macname{detokenizespacetok} macro for handling
+ implicit and active-implicit spaces}
+\newcommand\detokenizespacetok[1]{\tctestifcon{\ifimplicittok}%
+ {\tctestifcon{\ifactivetok}{\theactivespace}{\string#1}}{\detokenize{#1}}}
+\end{exampleC}
+\medskip
+
%%%
\resettokcycle
\Characterdirective{\addcytoks{\underdot{#1}}}
-\begin{exampleA}[lefthand width=10cm]{Implicit spaces work{,} too}
+\begin{exampleA}[lefthand width=10cm]{Implicit spaces also work}
\resettokcycle
\Characterdirective{\addcytoks{\underdot{#1}}}
-\def\:{\let\mysptoken= } \: %
+\def\:{\let\z= } \: %
\catcode`\_=10 %
\catcode`\ =12 %
\tokencyclexpress
-\fbox{a\mysptoken{}c d} b_g\itshape f\upshape
+\fbox{a\z{}c d} b_g\itshape f\upshape
\endtokencyclexpress
\end{exampleA}
+\begin{exampleA}[lefthand width=10cm]{Active Implicit spaces work{,} too}
+\resettokcycle
+\Characterdirective{\addcytoks{\underdot{#1}}}
+\def\:{\let\z= } \: %
+\catcode`Q=\active %
+\def\:{\let Q= } \: %
+\catcode`\_=10 %
+
+\tokencyclexpress
+xQx x_x\z{}x%
+\endtokencyclexpress
+
+\resettokcycle
+\Spacedirective{\addcytoks{\textcolor{cyan}\bgroup}%
+ \addcytoks[x]{(\detokenizespacetok{#1})}%
+ \addcytoks{\egroup}}
+
+\medskip
+\tokencyclexpress
+xQx x_x\z{}x%
+\endtokencyclexpress
+\end{exampleA}
+
\endgroup
Modified: trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex
===================================================================
--- trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex 2020-10-01 20:42:55 UTC (rev 56500)
+++ trunk/Master/texmf-dist/tex/generic/tokcycle/tokcycle.tex 2020-10-01 21:05:19 UTC (rev 56501)
@@ -1,7 +1,7 @@
\def\tcname {tokcycle}
-\def\tcver {1.12}
+\def\tcver {1.2}
%
-\def\tcdate {2020/02/11}
+\def\tcdate {2020/10/01}
%
% Author : Steven B Segletes, Christian Tellechea (contributor)
% Maintainer : Steven B Segletes
@@ -57,7 +57,8 @@
\def\addtc at depth{\advance\tc at depth 1}
\def\subtc at depth{\tc at depth=\numexpr\tc at depth-1\relax}
\def\tc at resetifs{\activetokfalse\implicittokfalse\tc at implicitgrpfalse
- \catSIXfalse\activecharfalse\activetokunexpandablefalse}
+ \catSIXfalse\activecharfalse\activetokunexpandablefalse
+ \def\theactivespace{}}
\long\def\count at stringtoks#1{\tc at earg\count at toks{\string#1}}
\long\def\count at toks#1{\the\numexpr-1\count@@toks#1.\tc at endcnt}
\long\def\count@@toks#1#2\tc at endcnt{+1\tc at ifempty{#2}{\relax}{\count@@toks#2\tc at endcnt}}
@@ -92,7 +93,7 @@
\def\endtokcycraw{\subtc at depth\tctestifnum{\tc at depth=0}{\@aftertokcycle}{}}
% LOOP ENTRY POINT
\def\@tokcycle{\tc at resetifs\futurelet\tc at next\detect at CANTabsorb}
-\def\detect at CANTabsorb{\tctestifx{\tc at next\tc at sptoken}{\stringify\@@@@@spcT}%
+\def\detect at CANTabsorb{\tctestifx{\tc at next\tc at sptoken}{\stringify\@@@@@@spcT}%
{\tctestifx{\tc at next\bgroup}{\stringify\@@@@grpT}{\can at absorb}}}
% NON cat1,10 TOKENS
\long\def\can at absorb#1{\tc at tok{#1}\trapcatSIX{#1}\expandafter\can at absorb@
@@ -103,12 +104,16 @@
\long\def\can at absorb@@#1{\let\@tmp=#1\test at ifmacro\@tmp{\implicittokfalse
\@macT#1}{\trapimplicitegrp#1\implicitgrpfork#1}\@tokcycle}
%CONVERT NEXT (SPACE OR BEGIN-GROUP) TOKEN TO STRING
-\def\stringify#1{\expandafter#1\string}% #1 WILL BE \@@@@@spcT or \@@@@grpT
+\def\stringify#1{\expandafter#1\string}% #1 WILL BE \@@@@@@spcT or \@@@@grpT
%SPACE DECODE
-\def\@@@@@spcT{\futurelet\tc at str\@@@@spcT}
-\def\@@@@spcT{\tctestifx{\tc at str\tc at sptoken}%
+\def\@@@@@@spcT{\futurelet\tc at str\@@@@@spcT}
+\def\@@@@@spcT{%
+ \tctestifx{\tc at str\tc at sptoken}%
{\def\@tmp{\@@spcT{ }}\expandafter\@tmp\tc at absorbSpace}% EXPLICIT SPACE
- {\implicittoktrue\expandafter\@@@spcT\tc at gobble}}% IMPLICIT SPACE
+ {\implicittoktrue\tctestifcon{\if\expandafter\@firstoftwo\string\\\tc at str}%
+ {\expandafter\@@@spcT\tc at gobble}% IMPLICIT MACRO SPACE
+ {\activetoktrue\@@@@spcT}}}% IMPLICIT ACTIVE SPACE
+\def\@@@@spcT#1{\def\theactivespace{#1}\trapactivechar{#1}\@@spcT{\tc at sptoken}}
\def\@@@spcT{\csmk{\expandafter\@@spcT\thecs}}
\def\@@spcT#1{\@spcT{#1}\@tokcycle}
% GROUP DECODE
@@ -223,4 +228,12 @@
- Documentation correction: \tokcycleenvironment, not \tokencycleenvironment
- Documentation correction: misspelling in tokcycle-examples.tex
- Redefined \tc at defx and \tc at earg to omit #2 as part of definition
- - Corrected \trapcatSIXb definition to account for revised \tc at earg definition.
\ No newline at end of file
+ - Corrected \trapcatSIXb definition to account for revised \tc at earg definition.
+
+v1.2 2020/10/1
+ -Added/fixed capability to handle active-implicit spaces. While the
+ #1 passed to the \Spacedirective, in such a case, is an implicit
+ space \tc at sptoken, the name of the active character from whence it
+ originated in the input stream is stored as an explicit cat-12 token
+ in the definition \theactivespace. The flags \implicittok and \activetok
+ are both set true, and the \activechar flag is checked, as well.
\ No newline at end of file
More information about the tex-live-commits
mailing list.