[OS X TeX] TeXShop, perl and utf-8 encoding problems

[Jonathan Kew]

On 20 Feb 2005, at 5:28 pm, Stephen Moye wrote:

>
> On Feb 20, 2005, at 8:13 AM, Jonathan Kew wrote:
>
>> On 20 Feb 2005, at 10:43 am, Stephen Moye wrote:
>>
>>> I have written a perl script that converts TeX-style accents (eg 
>>> \"o) to their equivalents (ö). This is to convert a large number of 
>>> Textures documents for use with TeXShop/XeTeX.
>>>
>>> Most of the accents come through when TeXed by TeXShop/XeTeX, but 
>>> there are some problems such as ï and ç which do not appear in the 
>>> TeXShop/XeTeX PDF -- they are completely missing.
>>>
>>> <snip>
>>
>> If I'm understanding the problem, it sounds as though Perl may not be 
>> writing the output as Unicode (UTF-8). Are you specifically 
>> requesting this in your script? E.g., with an open() call such as:
>> 	open(OUTFILE, ">:utf8", $outFileName);
>> or by using the binmode() function to set the encoding layer of the 
>> filehandle, once it's open.
>
> Thanks for that tip. I'm still quite new to perl. Sadly, however, the 
> output is producing human-unreadable TeX files and incorrect output 
> when TeXed.

One possible scenario I can imagine: if the Perl script is now writing 
correct UTF-8 Unicode, but TeXShop is assuming a different encoding 
when it opens the file, then yes, it will appear human-unreadable. And 
if you then add an "%&encoding=UTF-8 Unicode" line at the top and 
save/typeset, TeXShop will now save the incorrectly-interpreted 
characters in their UTF-8 form, so you'll also get garbage when you 
typeset.

To prevent this, make sure that: (a) the Perl script puts an 
"%&encoding=UTF-8 Unicode" line at (or near) the beginning of its 
output, so that TeXShop interprets the file correctly; or (b) the 
TeXShop default encoding is set to UTF-8 in Preferences; or (c) you use 
the Open command and explicitly choose UTF-8 encoding when opening the 
file.

I'd be happy to take a look and see if I can offer any more insight, if 
you'd like to send me the Perl script (and preferably a small example 
file) off-list.

JK
--------------------- Info ---------------------
Mac-TeX Website: http://www.esm.psu.edu/mac-tex/
           & FAQ: http://latex.yauh.de/faq/
TeX FAQ: http://www.tex.ac.uk/faq
List Post: <mailto:MacOSX-TeX at email.esm.psu.edu>