strangeness of floating point math

Question

I am using JMP15.

I recently realized that a script I've used for some time had a small error in it that stems from a floating point quirk which is illustrated below.

The first two lines calculate the same number -> 23.

When I use the ceiling function, the first version returns 24 and the second returns 23.

Should this be happening?

show(((1104/2016)*42));
show(((42*1104/2016)));
show(ceiling((1104/2016)*42));
show(ceiling((42*1104/2016)));

/*:

(1104 / 2016) * 42 = 23;
42 * 1104 / 2016 = 23;
Ceiling((1104 / 2016) * 42) = 24;
Ceiling(42 * 1104 / 2016) = 23;

Byron_JMP · Answer

I think I'm going to frame that answer

Craige_Hales · Answer

2016: 2×2×2×2×2×3×3×7

1104: 2×2×2×2×3×23

42: 2×3×7

1104/2016 is (2×2×2×2×3×23) / (2×2×2×2×2×3×3×7)

23/(2×3×7) is a repeating fraction in base 10 AND base 2 (hex, base 16, shown with 4 bits per hex digit):

0.5476190476190476190476190... or 3FE186186186186... (lucky 12-bit repeat for visualization!)

Both of those are truncated, and imprecise. The actual value for the hex base 2 representation is rounded to 3FE1861861861862, which is slightly high (3FE is the exponent). Multiply by 42, after the divide, is also slightly high.

(42×1104)/2016 is  ((2×3×7)×(2×2×2×2×3×23)) / (2×2×2×2×2×3×3×7)

Multiply by 42, ahead of the divide, gets nice reduction of the prime divisors.

Dan_Obermiller · Answer

You have already seen that this is a floating point quirk. Should it be happening, technically no, but there will always be strange "rounding issues" due to floating point calculations.

If you put the first calculation into a column (1104/2016)*42, change the width of the column to be 30 (arbitrarily large), format to Fixed Decimal with 20 decimal places you will see that the number is stored as: 23.00000000000000355271.