[LaTeX] beamer and inputenc (utf8x) issue

[Don Hosek]

I suppose, one could convert the file to NFD normalization. Then á would be represented by the sequence a + ´ and then you would grab a rather than á. I’ve actually spent a bunch of time implementing unicode segmentation for finl (the existing Rust code for segmentation didn’t have the right interface for my needs). Aside from UTF-8 multi-byte sequences (which is anything not in 7-bit ASCII), there are all the assorted combining characters which include all manner of combining characters not to mention combining Jamo for Hangul, multi-character glyphs like country and region flags and some Emoji built by combining base characters using Unicode ZWJ. 

-dh

> On 1 Jul 2022, at 14:56, Dr Nicola L C Talbot via tex-live <tex-live at tug.org> wrote:
> 
>>>> If your input is in UTF8, it is better to use an engine working
>>>> internaly in Unicode, i.e luatex or xetex.
>>> 
>>> Millions of documents use utf8 with pdflatex without problems. I run
>>> 95% of my documents with pdflatex. They are all utf8-encode and as
>>> I'm german my texts do contain umlauts and other non-ascii chars.
>>> The only thing that pdflatex can't handle are combining accents.
>>> 
>>> 
>> I am not that lucky, most of my old pdflatex documents fail. I often
>> have \everypar containing a macro with one parameter for setting an
>> initial. If I output it as {\otherfont #1} and the token is the first
>> octet of a multioctet character, it fails. Character V as an initial
>> needs an extra kern thus if the macro contains \if#1V and #1 is the
>> first octet of a multioctet character, it fails. I often use
>> \futurelet\testchar\dosomething and if \testchar becomes the first
>> octet of a multioctet character, \dosomething fails. And it happens
>> even without hyperref. I stopped using pdflatex a few years ago. Now I
>> have 15 versions of TeX Live installed and when I have to recompile an
>> old document, I go back in history and try, in which version of TL the
>> document works. It is quite common for me that he old pdflatex
>> documents do not work in the current TL.
>> Your documents were presumably not specifying an encoding.
>> Since the default encoding was switched to UTF-8 we have had essentially no reports of documents breaking.
>> Any document that was correctly declaring  its encoding continues to work the same way, and any old document
>> using non-ascii characters without declaring an encoding (which was possible but never supported and produced
>> good or bad results depending on the font encoding in use) can be used with a current latex by adding
>> \UseRawInputEncoding
> 
> I think the point regarding pdflatex vs a Unicode engine is pertinent for the particular cases where a multioctet character needs to be grabbed in its entirety without explicitly grouping it. This has always been an issue with utf8 and inputenc both now and historically.
> 
> For example, to sentence case some text:
> 
> \MakeUppercase ábc
> 
> This works without a problem with xelatex and lualatex but fails with pdflatex. This is a problem for glossaries.sty where I have to tell users they need to group the first letter if they want to use any sentence casing commands. This isn't intuitively obvious since visually the character looks like a single entity.
> 
> Example document:
> 
> \documentclass{article}
> 
> \usepackage{glossaries}
> 
> \newglossaryentry{elite}{name={élite},description={...}}
> 
> \begin{document}
> \Gls{elite} forces entered the building.
> \end{document}
> 
> Again, it works fine with xelatex and lualatex but not with pdflatex, which requires:
> 
> \newglossaryentry{elite}{name={{é}lite},description={}}
> 
> (Incidentally, UTF-8 now works in glossary labels. It's the sentence-casing that's the issue here.)
> 
> Some time ago I experimented with trying to grab multiple octets (in datatool-base.sty, which is used by glossaries.sty), which used to work for two-byte octets but it's stopped working now. I haven't had time to investigate, but it would be really useful if inputenc provided a way to grab all octets of a multioctet character so that sentence casing (and similar tasks) can work without having to group the initial character.
> 
> Regards
> Nicola Talbot