[luatex] fio library byte order

[Reinhard Kotucha]

On 2020-06-27 at 09:59:43 +0200, Hans Hagen wrote:

 > On 6/27/2020 2:56 AM, Reinhard Kotucha wrote:
 > > On 2020-06-20 at 10:25:33 +0200, Hans Hagen wrote:
 > >
 > > The host byte order must always be determined automatically,
 > > either with Luigi's approach or probably more easily with
 > > ntohs(3) if this function is available on Windows too.  The file
 > > byte order has to be specified by the user because it depends on
 > > the file format.
 >
 > the lib is meant for usage in known scenarios (known, documented
 > file formats), not arbitrary, depending on architecture or
 > implementation

Sure.  This is what I said.  The file byte order has to be set by the
user, one way or another.  The host (system) byte order must be
determined at compile time.  Otherwise it's impossible to write
portable scripts.

 > (btw, the format file used to normalize to hig endian but that was
 > dropped long ago already: formats are no longer portable, which in fact
 > was already dropped before that)

I don't understand.  All format files in TeX Live work on all systems.
They have a distinct byte order and are portable among all systems
supported by TeX Live.  What do you mean if you say "formats are no
longer portable"?

 > > If a particular file format has a BOM in its header, the BOM can
 > > be evaluated by the user, for instance with fio.readline().  This
 > > means that a user should be able to specify the andianness at any
 > > time, not necessarily in advance.
 >
 > sure but a few extra readers would solve that

 > > As far as I understand it's sufficient that the relevant
 > > functions read{cardinal,integer}{2,4} obey a flag which tells
 > > them whether byte re-ordering is necessary.  The flag has to be
 > > set if host and file byte orders are different.  I don't know
 > > whether we have to consider 64 bit integers too.
 >
 > that adds passing parameters and checking them for each call
 > ... you can then as well use lua's 'read' function and convert with
 > string.byte/char which is then about equally fast

This is what I actually did.  It took 14 s to process a PNM file, way
too much if I have to process hundreds or thousands files.  I ported
the script to C and could process the file within 270 ms.  I can't
imagine that obeying a variable in C can slow down everything so much.

I'm not very familiar with C programming.  You say that it's expensive
to pass arguments to a function.  What I had in mind is that functions
obey a global variable at runtime which denotes whether byte order
conversion is necessary or not.

 > > If you intend to go this way the number of functions in
 > > liolibext.c can be halved because there is no significant
 > > difference between a buffer and a string.  Only very few
 > > functions have to be aware of endianness.
 >
 > halved in calls to simple functions, enlarged by more checking
 > .. .more pain than gain

It's just a suggestion.  For me, as a user, it doesn't matter how
things work behind the scenes.


Regards,
  Reinhard

--
------------------------------------------------------------------
Reinhard Kotucha                            Phone: +49-511-3373112
Marschnerstr. 25
D-30167 Hannover                    mailto:reinhard.kotucha at web.de
------------------------------------------------------------------