strangeness of floating point mathI am using JMP15.
I recently realized that a script I've used for some time had a small error in it that stems from a floating point quirk which is illustrated below.
The first two lines calculate the same number -> 23.
When I use the ceiling function, the first version returns 24 and the second returns 23.
Should this be happening?
show(((1104/2016)*42));
show(((42*1104/2016)));
show(ceiling((1104/2016)*42));
show(ceiling((42*1104/2016)));
/*:
(1104 / 2016) * 42 = 23;
42 * 1104 / 2016 = 23;
Ceiling((1104 / 2016) * 42) = 24;
Ceiling(42 * 1104 / 2016) = 23;
ScriptingRe: strangeness of floating point mathYou have already seen that this is a floating point quirk. Should it be happening, technically no, but there will always be strange "rounding issues" due to floating point calculations.
If you put the first calculation into a column (1104/2016)*42, change the width of the column to be 30 (arbitrarily large), format to Fixed Decimal with 20 decimal places you will see that the number is stored as: 23.00000000000000355271.
Re: strangeness of floating point math2016: 2×2×2×2×2×3×3×7
1104: 2×2×2×2×3×23
42: 2×3×7
1104/2016 is (2×2×2×2×3×23) / (2×2×2×2×2×3×3×7)
23/(2×3×7) is a repeating fraction in base 10 AND base 2 (hex, base 16, shown with 4 bits per hex digit):
0.5476190476190476190476190... or 3FE186186186186... (lucky 12-bit repeat for visualization!)
Both of those are truncated, and imprecise. The actual value for the hex base 2 representation is rounded to 3FE1861861861862, which is slightly high (3FE is the exponent). Multiply by 42, after the divide, is also slightly high.
(42×1104)/2016 is ((2×3×7)×(2×2×2×2×3×23)) / (2×2×2×2×2×3×3×7)
Multiply by 42, ahead of the divide, gets nice reduction of the prime divisors.
Re: strangeness of floating point mathI think I'm going to frame that answer
