Dear Community,
we are processing large tables and want to exclude outliers.
We found, that the platform robust fit outlier according to huber excludes all rows, unless in the summary there are found no outliers, see picture and sample table script to reproduce (JMP15 or JMP16 on Windows).
Does this really make sense or is there a bug, I would expect a different behaviour.
New Table( "Huber-outlier-0-sigma",
Add Rows( 100 ),
New Script(
"Explore Outliers of value",
Explore Outliers( Y( :value ), Robust Fit Outliers )
),
New Column( "value",
Numeric,
"Continuous",
Format( "Best", 12 ),
Set Values(
[0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4,
0.4, 0.4]
),
Set Display Width( 83 )
),
)
Please contact support@jmp.com to report the problem.
I cannot assess the impact of this oddness to user's production environment. And I cannot speak for the mind of the developer, whether there is a solution.
Here is my view from a numerical angle. In this case, all data are outliers indeed, based on how the outliers are defined and how our computer calculates. The root lies in the floating point arithmetic. There are two issues with this case:
- First 0.4 cannot be exactly represented in computer. Imagine 0.4 in computer is 0.40000000000000xx, with some tiny appendage in the end.
- Second, Huber Center estimate is not 0.4 exactly either. Even the mean of a bunch of 0.4 is not 0.4 in computer, due to floating point arithmetic as well. Imagine here the Huber Center estimate is 0.40000000000000yy, with some other tiny appendage yy in the end, but different from xx.
Here is the screenshot after I change the format to Fixed with enough length for digits after the decimal place. You can see the appendage in the Huber Center.
Given the estimate of Huber Spread is zero and all observations are numerically different from the Huber Center, they are indeed light years away from the Huber Center, so they are outliers according to the definition and how they are calculated in our imperfect computer.
