<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi David,<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 21 Feb 2021, at 11:02 pm, David Carlisle <<a href="mailto:d.p.carlisle@gmail.com" class="">d.p.carlisle@gmail.com</a>> wrote:</div>
</blockquote>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class="">I don't think there is any reasonable way to say you can comment out parts of a file in a different encoding.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">I’m not convinced that this ought to be correct for TeX-based software.</div>
<div class=""><br class="">
</div>
<div class="">TeX (not necessarily XeTeX) has always operated as a finite-state machine.</div>
<div class="">It *should* be possible to say that this part is encoded as such-and-such,</div>
<div class="">and a later part encoded differently.</div>
<div class=""><br class="">
</div>
<div class="">I fully understand that editor software external to TeX might well have difficulties </div>
<div class="">with files that mix encodings this way, but TeX itself has always been byte-based </div>
<div class="">and should remain that way.</div>
<div class=""><br class="">
</div>
<div class="">A comment character is meant to be viewed as saying that:</div>
<div class=""> *everything else on this line is to be ignored*</div>
<div class="">– that’s the impression given by TeX documentation.</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">But you only know it is a comment character if you can interpret the incoming byte stream
<br class="">
</div>
<div class="">If there are encoding errors in that byte stream then everything ls is guess work.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Who said anything about errors in the byte stream?</div>
<div>Once you have encountered the (correct) comment character, </div>
<div>what follows on the rest of the line is going to be discarded, </div>
<div>so its encoding is surely irrelevant.</div>
<div><br class="">
</div>
<div>Why should the whole line need to be fully tokenised,</div>
<div>before the decision is taken as to what part of it is retained?</div>
<div><br class="">
</div>
In the case of a package file, rather than author input for typesetting,</div>
<div>the intention of the coding is completely unknown, </div>
<div>is probably all ASCII anyway, except (as in this case) for comments intended </div>
<div>for human eyes only, following a properly declared comment-character.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<div class=""><br class="">
</div>
<div class="">In this particular case with mostly ascii text and a few latin-1 characters it may be that you can guess that</div>
<div class="">the invalid utf-8 is in fact valid latin1 and interpret it that way,
</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>You don’t need to interpret it as anything; that part is to be discarded.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<div class="">and the guess would be right for this file</div>
<div class="">but what if the non-utf8 file were utf-16 or latin-2 or </div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Surely the line-end characters are already known, and the bits&bytes </div>
<div>have been read up to that point *before* tokenisation.</div>
<div>So provided the tokenisation of the comment character has occurred before</div>
<div>tackling what comes after it, why would there be a problem?</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<div class="">... just guessing the encoding (which means guessing where the line and so the comment ends)</div>
<div class="">is just guesswork.<br class="">
</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>No guesswork intended.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class="">The file encoding specifies the byte stream interpretation before any tex tokenization</div>
<div class="">If the file can not be interpreted as utf-8 then it can't be interpreted at all.
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Why not?</div>
<div class="">Why can you not have a macro — presumably best on a single line by itself –</div>
</div>
</div>
</blockquote>
<div class=""> </div>
<div class="">there is an xetex primitive that switches the encoding as Jonathan showed, but guessing a different encoding</div>
<div class="">if a file fails to decode properly against a specified encoding is a dangerous game to play.</div>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I don’t think anyone is asking for that.</div>
<div><br class="">
</div>
<div>I can imagine situations where coding for packages that used to work well </div>
<div>without UTF-8 may well be commented involving non-UTF-8 characters.</div>
<div>(Indeed, there could even be binary bit-mapped images within comment sections;</div>
<div>having bytes not intended to represent any characters at all, in any encoding.)</div>
<div><br class="">
</div>
<div>If such files are now subjected to constraints that formerly did not exist,</div>
<div>then this is surely not a good thing.</div>
<div><br class="">
</div>
<div><br class="">
</div>
<div>Besides, not all the information required to build PDFs need be related to</div>
<div>putting characters onscreen, through the typesetting engine.</div>
<div><br class="">
</div>
<div>For example, when building fully-tagged PDFs, there can easily be more information</div>
<div>overall within the tagging (both structure and content) than in the visual content itself. </div>
<div>Thank goodness for Heiko’s packages that allow for re-encoding strings between</div>
<div>different formats that are valid for inclusion within parts of a PDF.</div>
<div><br class="">
</div>
<div>I’m thinking here about how a section-title appears in:</div>
<div> bookmarks, ToC entries, tag-titles, /Alt strings, annotation text for hyperlinking, etc.</div>
<div>as well as visually typeset for on-screen.</div>
<div>These different representations need to be either derivable from a common source,</div>
<div>or passed in as extra information, encoded appropriately (and not necessarily UTF-8).</div>
<div><br class="">
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<div class="">So I don't think such a switch should be automatic to avoid reporting encoding errors.<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I reported the issue at xstring here</div>
<div class=""><a href="https://framagit.org/unbonpetit/xstring/-/issues/4" class="">https://framagit.org/unbonpetit/xstring/-/issues/4</a></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">David</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">
<div class="">
<div class="">that says what follows next is to be interpreted in a different way to what came previously?</div>
<div class="">Until the next switch that returns to UTF-8 or whatever?</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">If XeTeX is based on eTeX, then this should be possible in that setting.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="">Even replacing by U+FFFD <br class="">
</div>
<div class="">is being lenient.</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>Why has the mouth not realised that this information is to be discarded?</div>
<div>Then no replacement is required at all.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class="">David</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<br class="">
<div class=""><br class="">
Dr Ross Moore<br class="">
Department of Mathematics and Statistics
<div class="">12 Wally’s Walk, Level 7, Room 734<br class="">
Macquarie University, NSW 2109, Australia<br class="">
T: +61 2 9850 8955 | F: +61 2 9850 8114<br class="">
M:+61 407 288 255 | E: <a href="mailto:ross.moore@mq.edu.au" class="">ross.moore@mq.edu.au</a><br class="">
<a href="http://www.maths.mq.edu.au" class="">http://www.maths.mq.edu.au</a><span style="font-size: 12px; line-height: normal;"><a href="http://mq.edu.au/" target="_blank" style="font-size: 12px; line-height: normal;" class=""><span><br class="Apple-interchange-newline" style="caret-color: rgb(0, 105, 217); color: rgb(0, 105, 217); font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-family: Arial, sans-serif; orphans: 2; widows: 2;">
<span style="caret-color: rgb(0, 105, 217); color: rgb(0, 105, 217); font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-family: Arial, sans-serif; orphans: 2; widows: 2;"><span><span><span><span><img apple-inline="yes" id="7DE63DA0-1462-49DB-A9FE-D8BCC28EC620" src="cid:image001.png@01D030BE.D37A46F0" class=""></span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">CRICOS
Provider Number 00002J. Think before you</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">print. </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">Please
consider the environment before printing this</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">email.</span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">This
message is intended for the addressee named</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">and
may </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">contain
confidential information. If you are not the</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">intended </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">recipient,
please delete it and notify the sender. Views</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">expressed </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">in
this message are those of the individual sender, and</span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class=""> </span><span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">are
not </span><br style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none;" class="">
<span style="font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; text-decoration: none; float: none; display: inline !important;" class="">necessarily
the views of Macquarie University.</span> </span></span></span></span></span></a></span></div>
<a href="http://mq.edu.au/" target="_blank" style="font-size: 12px; line-height: normal;" class=""></a></div>
<br class="">
</body>
</html>