Re: Round-off errors in metrics & fontinst/finst status...

[Rebecca and Rowland <rebecca@astrid.u-net.com>]

>At 04:36 PM 97/12/17 -0800, Melissa O'Neill wrote:
>
[snip]
>How serious is this?  Well, the error of rounding obviously lies
>uniformly distributed in the [-0.5, 0.5] range.

Not obvious: does the software really round, or does it truncate?  (I'll
assume it does proper rounding, because there's no excuse.  But some
programmers are very lazy.)

>  The average character
>width in CMR10 is 586 (on the 1000 per em scale).  So the error is
>of the order of +/- 0.1%   In 150mm we may see in the worst case
>+/- 0.15mm.  Is that noticeable?  Probably.  Of course in the typical
>situations lots of positive errors of varying size are added to lots of
>negative errors of various size and so the overall error is less.

Assuming that the error is evenly distributed, the error will be *very*
much less.  It'd take some horrible stats to work out what you could expect
normally - I'd guess that one order of magnitude less is about right.

So the typical error would be about 15 microns in 150 mm, which probably
isn't noticeable.

>The time it does add up systematically is when one is using the same
>character repeatedly (as on a fill) or in a fixed width font.  Note that the
>error can be worse than 0.15mm if the character is narrower than 586.

And it can be less if the rounding is less than 0.5 - this 0.15mm is
more-or-less worst-case.  The rms rounding error is something like 0.3, so
the typical error with a repeated character will be nearer 0.05%, or 75
microns in 150mm; typical error with random characters will be nearer 7.5
microns.  I'm not inclined to fret over errors of less than 0.1mm across a
page.

[snip]

Rowland.