rounding problems

[Pierre MacKay]

  > What amazes me about TeX is that someone like Knuth could write something
  > that's as bad at arithmetic as TeX is.

Wow.  Have you tried to think back to all the forms of floating-point
representation that were there in the late 70's.  I don't know what the
status of the IEEE format was then, but I can be pretty confident that
even if it was already taking shape, it was certainly not reliably
available on a range of systems.  Moreover, there was the special problem
of the 36-bit word in the processors used by the Dec10 and the Dec20.  
(They were pretty terrific machines in their day, and the Dec20 inspired
Tops20, which is the only archaic operating system that still raises
twinges of regret at its disappearance.  

Knuth insisted on integer-only arithmetic for the precise reason that
he did not trust the floating-point behaviour even of his much-loved
Dec10.  The infinitesimal Scaled Point, which you can always get at if
you really need it, provides access to enough precision for anything
that really needs to be done.  Yes, it is a bit harder to do floating-point
calculations that way, but you know that the rounding errors are limited
to what can be expressed in scaled points if you do it right.  Consider
the extraordinary job done by PicTeX.  TeX log files do express certain
values in floating-point notation, and the minor variants in trip and
trap log files show that floating point can still be an uncertain
format when real precision is needed.  But TeX never does the calculation
in floating-point, only the expression of the results.  

I find it hard to believe that the precision of scaled points is ever
to coarse to solve any reasonable typesetting problem.  Integer arithmetic
always involves the risk of overflow, either into a pseudo-negative number
(bad underlying code) or an arbitrary truncation.  Knuth has showed how
to tame that problem in Web coding in dvitype and numerous other places.
You sometimes have to cascade calculations to avoid overflow, but at
the speed of present-day processors, so what?  I do calculations to three
and four decimal places in TeX, and it sometimes means taking four steps
for something that a floating point processor would do in one.  But I know
that the four decimal places will be the same on every processor if I
accept that slight extra effort.

Email:  mackay@cs.washington.edu		Pierre A. MacKay
Smail:  Department of Classics			Emeritus Druid for
	218 Denny Hall, Box 353110		Unix-flavored TeX
	University of Washington
	Seattle, WA 98195
	(206) 543-2268 (Message recorder)