<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Ulrike,<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 22 Feb 2021, at 7:52 am, Ulrike Fischer <<a href="mailto:news3@nililand.de" class="">news3@nililand.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Am Sun, 21 Feb 2021 20:26:04 +0000 schrieb Ross Moore:<br class="">
<br class="">
> Once you have encountered the (correct) comment character,<br class="">
> what follows on the rest of the line is going to be discarded,<br class="">
> so its encoding is surely irrelevant.<br class="">
> <br class="">
> Why should the whole line need to be fully tokenised,<br class="">
> before the decision is taken as to what part of it is retained?<br class="">
<br class="">
Well you need to find the end of the line to know where to stop with<br class="">
the discarding don't you? So you need to inspect the part after the<br class="">
comment char until you find something that says "newline”.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>My understanding is that this *is* done first.</div>
<div>Similarly to TeX's \read to <csname> which grabs a line of input from a file, </div>
<div>before doing the tokenisation and storing the result in the <csname>.</div>
<div> page 217 of The TeXbook</div>
<div><br class="">
</div>
<div>If I’m wrong with this, for high-speed input, then yes you need to know where to stop.</div>
<div>But that’s just as easy, since you stop when a byte is to be tokenised</div>
<div>as an end-of-line character, and these are known. </div>
<div>You need this anyway, even when you have tokenised every byte.</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div>So all we are saying is that when handling the bytes between</div>
<div>a comment and its end-of-line, just be a bit more careful.</div>
<div><br class="">
</div>
<div>It’s not necessary for each byte to be tokenised as valid for UTF-8.</div>
<div>Maybe change the (Warning) message when you know that you are within</div>
<div>such a comment, to say so. That would be more meaningful to a package-writer, </div>
<div>and to an author who uses the package, looks in the .log file, and sees the message.</div>
<div><br class="">
</div>
<div>None of this is changing how the file is ultimately processed;</div>
<div>it’s just about being friendlier in the human interface.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""><br class="">
<br class="">
<br class="">
-- <br class="">
Ulrike Fischer <br class="">
<a href="https://www.troubleshooting-tex.de" class="">https://www.troubleshooting-tex.de/</a><br class="">
</div>
</div>
</blockquote>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
All the best.
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>Ross</div>
<div class=""><br class="">
<div class=""><br class="">
Dr Ross Moore<br class="">
Department of Mathematics and Statistics
<div class="">12 Wally’s Walk, Level 7, Room 734<br class="">
Macquarie University, NSW 2109, Australia<br class="">
T: +61 2 9850 8955 | F: +61 2 9850 8114<br class="">
M:+61 407 288 255 | E: <a href="mailto:ross.moore@mq.edu.au" class="">ross.moore@mq.edu.au</a><br class="">
<a href="http://www.maths.mq.edu.au" class="">http://www.maths.mq.edu.au</a><span style="font-size: 12px; line-height: normal;"><a href="http://mq.edu.au/" target="_blank" style="font-size: 12px; line-height: normal;" class=""><span><br class="Apple-interchange-newline" style="caret-color: rgb(0, 105, 217); color: rgb(0, 105, 217); font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-family: Arial, sans-serif; orphans: 2; widows: 2;">
<span style="caret-color: rgb(0, 105, 217); color: rgb(0, 105, 217); font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-family: Arial, sans-serif; orphans: 2; widows: 2;"><span><span><span><span><img apple-inline="yes" id="28D74877-9EA2-4030-BC1F-30478F0C28CE" src="cid:image001.png@01D030BE.D37A46F0" class=""></span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">CRICOS
Provider Number 00002J. Think before you</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">print. </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">Please
consider the environment before printing this</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">email.</span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">This
message is intended for the addressee named</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">and
may </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">contain
confidential information. If you are not the</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">intended </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">recipient,
please delete it and notify the sender. Views</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">expressed </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">in
this message are those of the individual sender, and</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">are
not </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">necessarily
the views of Macquarie University.</span> </span></span></span></span></span></a></span></div>
<a href="http://mq.edu.au/" target="_blank" style="font-size: 12px; line-height: normal;" class=""></a></div>
<br class="">
</div>
</body>
</html>