[pdftex] pdflatex failures on big-endian architectures (s390x / ppc64)

[Hilmar Preuße]

Dear pdfTeX developers,

This is rather 1:1 copy of the Debian bug report #1054218. I don't 
attach the original sample files, but rather some minimal examples I 
generated on ppc64. Interestingly the issue does not occur, when using 
plain TeX, I needed to construct a LaTeX example.

Further I got the information, that it worked fine on Debian oldstable, 
meanwhile it does not work since Debian stable. Debian unstable is still 
affected. I did not find anything in 
texlive-source/texk/web2c/pdftexdir/ChangeLog, which points to a bug fix 
for this.

hille at debian-haka2:~$ pdftex --version
pdfTeX 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian)
kpathsea version 6.3.2
Copyright 2020 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.37; using libpng 1.6.37
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.02

Stopped working:

hille at haka2:~$ pdftex --version
pdfTeX 3.141592653-2.6-1.40.24 (TeX Live 2022/Debian)
kpathsea version 6.3.4
Copyright 2022 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.39; using libpng 1.6.39
Compiled with zlib 1.2.13; using zlib 1.2.13
Compiled with xpdf version 4.04


<snip>
The unittests of the 'plastex' package run pdflatex to generate some
figures, and then extract the text from the figures to verify that
various implementation details of the package are working. These tests
pass on all release architectures except s390x. They also fail on ppc64.
The common feature of the failures is that the architecture is
big-endian.

The failures are all similar to:

   AssertionError: 'hi' != '\x00\x00'

i.e. the text that is found in the PDF (either by gs or pdftotext) is
the same number of bytes as the original text, but is all \0. The
extraction is platform-independent — the attached 1054218_ppc64.pdf 
yields \0\0 for its text no matter what arch pdftotext or gs is run on.

The PDFs all _look_ OK in any PDF viewer, it's just the text extraction
that fails.

If the pdf is generated via latex followed by dvipdf then the extracted
text is correct (up to whitespace); if the pdf is generated by lualatex
then he extracted text is correct.

It seems that pdflatex is mishandling embedding the text on big endian
systems. Speculating wildly... it looks a bit like pdflatex is taking
the wrong byte out of a multibyte character representation, and ending
up with \0 rather than the byte of interest, but I don't know how
pdflatex is representing the characters internally or how it is encoding
them into the PDF.

Business impact:
While I don't expect that there are many direct users of pdflatex on 
s390x, testing migration within Debian now requires successful 
completion of unittests on s390x, and so arch-specific bugs on s390x 
become relevant.
<snip>

Hilmar
-- 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1054218_ppc64.tex
Type: text/x-tex
Size: 99 bytes
Desc: not available
URL: <https://tug.org/pipermail/pdftex/attachments/20231023/33f169f8/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1054218_ppc64.pdf
Type: application/pdf
Size: 11161 bytes
Desc: not available
URL: <https://tug.org/pipermail/pdftex/attachments/20231023/33f169f8/attachment-0002.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1054218_amd64.pdf
Type: application/pdf
Size: 11159 bytes
Desc: not available
URL: <https://tug.org/pipermail/pdftex/attachments/20231023/33f169f8/attachment-0003.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://tug.org/pipermail/pdftex/attachments/20231023/33f169f8/attachment-0001.sig>