nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

floating number error with a simple substraction

maurogerber

I cant say if JMP can substract two simple numbers correctly but I tried the following numbers:

16.11-15.12

0.99 (ok, so far so good)

15.29-15.18

0.109999999999999 (not so good any more)

15.29-15.19

0.0999999999999997 (a seven?)

So where is the hook I placed wrong?

This happens with JMP 12.2.0 (64-bit) in the script editor.

Accepted answers

Mark_Bailey

You might want to read this resource: JMP Numerical Accuracy You can find such white papers for other popular software such as spreadsheets. It is a know issue but generally not a problem.

XanGregg

Regarding "significant digits", you can get that by using the "Precision" format with the "Format" function. Also, the "Char" function supports two addition arguments, one for the width and one for the number of decimal places, if you just want to skip the Round operation for display.

Char( 15.29 - 15.19, 10, 2); // "0.10"
Format( 12.345, "Precision", 3); // "12.3"
Format( 1.2345, "Precision", 3); // "1.23"

Regarding the original issue, it boils down to the fact that computers store numbers in base 2 (binary) but display them in base 10, and some base 10 numerals like "0.1" have no exact representation in base 2.

All comments

Mark_Bailey

In mathematics, simple arithmetic would yield exact results in these differences: 15.29 - 15.18 = 0.11 and 15.29 - 15. 19 = 0.10. But the result in JMP is finite precision arithmetic in a computer. The result depends on how the real number is represented as a floating point number and how the arithmetic operators work.with this representation. It is not difficult to find cases where the result of the operator and the mathematical result do not agree.

JMP might use a standard floating point numeric software library or a dedicated hardware floating point processor for such operations.

There are some softwares that use extended or 'unlimited' precision (limited only by physical resources) but it generally doesn't change the practical results in our statistical methods, such as regression.

One solution in your case is to round the result in a formula or script:

Round( 15.2900000 - 15.1900000, 2 )

Then you get 0.1 for the result. Problem solved.

maurogerber

not quiet, to "exten" it a bit (and the reason why its bugging me) is that the amount of precision depend on user input

If he uses 15.29mm and 15.19mm he will get the correct 0.1mm with Round(x,2).

If he uses 0.01529m and 0.01519m he will get 0m and will probably stand in my door the seccand after he sees the result *g*

And I can't see if "significant digits" is supported like round or char.

Mark_Bailey

You might want to read this resource: JMP Numerical Accuracy You can find such white papers for other popular software such as spreadsheets. It is a know issue but generally not a problem.

Mark_Bailey

I understand. My example with the Round() function was only meant to illustrate one method for dealing with the limited precision of the floating point number and associated arithmetic.You can build richer solutions to handle more complex cases.

Did you use Help > Books > Scripting Guide or Help > Scripting Index to see the capabilities of the Round() or Char() functions?

Dan_Obermiller

Just to illustrate the issue is NOT a JMP issue, here is what happens in Excel.

X	Y	X - Y Formula
16.11	15.12	0.99000000000000000
15.29	15.18	0.10999999999999900
15.29	15.19	0.09999999999999960

maurogerber

THX for the inputs, small question, do you consider an option as "n significant digits" for show, char and round? that would help a lot.

Mark_Bailey

I am not sure what you are asking. If you are asking if these functions have the option to specify significant digits as well as the number of decimal places, then the answer is no. If you are asking if I would consider adding such an option, then that request is a matter for JMP Development.

XanGregg

Char( 15.29 - 15.19, 10, 2); // "0.10"
Format( 12.345, "Precision", 3); // "12.3"
Format( 1.2345, "Precision", 3); // "1.23"

maurogerber

I endet up using

format(mean(y),6)

so in the report a very small number gets converted into 1.574e-6 and don't use up a lot of space.

but

Format( 1.2345, "Precision", 3)

would be the way for "significant digits".