nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Possible to increase floating point number precision?

nikles

Hi. Is there a way to specify the floating point precision JSL uses when performing calculations? That is, the number of significant digits used. For instance, on my computer 1 minus 1e-15 equals 0.999... (15 "9"s). But if I try 1 minus 1e-16, I just get 1.

x = 1;

y = 1e-15;

z = x - y;

Show(z); //z = 0.999999999999999 (aka 15 decimal places)

y = 1e-16;

z = x - y;

Show(z); //z = 1

Thanks.

Accepted answers

txnelson

Here is the complete script that I ran to give you the results I posted.

x = 1;

y = 1e-15;

z = x - y;

Show(z); //z = 0.999999999999999 (aka 15 decimal places)

y = 1e-16;

z = x - y;

Show(format(z, "Fixed Dec", 30, 20));

I ran the script in 11.21.1 and got the results you reported. So the difference is the change to version 12.2.

All comments

txnelson

What you are seeing is JMP attempting to display the data in a default friendly format. The precision behind the value does not change when specifying a format to display the value in. So if you specify to display the value of "z" with more precision it will do that.

Show(format(z, "Fixed Dec", 30, 20)); = "0.99999999999999989000";

Craige_Hales

JMP uses the standard hardware floating point support. IEEE floating point - Wikipedia, the free encyclopedia

There are about 15 significant digits, as you've noticed.

JMP does not have support for longer representations. (Big Int, Big Num libraries are not in JMP at this time)

As txnelson points out, sometimes a default format for printing a number may not show all 15 or so possibly interesting digits.

nikles

Jim,

I'm sorry but that does not appear to work on my system. I used the format command and the answer still came out as "1". Obviously it works on your system, but not mine. My company still has us using JMP 11.2.1, if that matters. My computer is < 1 yr old though.

Question: the documentation for Format is a little confusing.

s = Format(x, formatstring, <currencycode>, <ndecimal>, < <<Use Locale(b=1)>)

Based on that, the "20" in your command occupies the currency code field, and the "30" occupies the ndecimal field. Naturally I care about the latter. What is the purpose of the "20"?

nikles

Hi again Jim. Also realized another important question. In your post you indicated the result to the problem 1 minus 1e-16 was:

0.99999999999999989000

(Note the "8" in the 16th decimal place)

Was this a typo?

txnelson

Here is the complete script that I ran to give you the results I posted.

x = 1;

y = 1e-15;

z = x - y;

Show(z); //z = 0.999999999999999 (aka 15 decimal places)

y = 1e-16;

z = x - y;

Show(format(z, "Fixed Dec", 30, 20));

I ran the script in 11.21.1 and got the results you reported. So the difference is the change to version 12.2.