outlier detection approach - iterative 3 sigmaWhat inspired this Wish List Request?
The aim is to filter the data and remove outliers in order to set aside wafers with many defects and identify the standard. the data follows a poisson distribution
Please describe the new or improved feature or solution you would like to see.
they would like to detect outlier by using iterative 3 sigmas approach.
it means:
- Measure mean and standard deviation of samples.
- Remove samples higher than mean plus 3 standard deviations.
- Repeat from Step 1 - for a total of 3 times.
What would the value be to you and/or other users if this new feature is provided?
Is this an external customer request?
STMicroelectronics - Crolles
Is there a competitive software package that has this feature or solution that we should be aware of?
Don't know
Will this impact a renewal, expansion, or adoption at a customer, and does this put an account at risk if we do not address this?
On-going ELA at STMicroelectronics
Data Blending and CleanupRe: outlier detection approach - iterative 3 sigmaHi @flo_kussener,
Would something like the below be something that could be adapted to the customer needs in the mean time? It is only a single pass for a defined standard deviation using std dev() function in JSL.
/*
This script was created by:
Chris Kirchberg, Principal Systems Engineer, JMP Statistical Discovery, LLC
Sam Garnder, Sr. Product Manager, JMP Statistical Discovery, LLC
Date: 2023-10-24
It is intended to show how one can filter/group/screen through thousands of columns in a wide data set to
help reduce a data set to useful or functional features/columns for further modeling/testing.
!!!!!!It was not evaluated by any QC/QA methods or departments. Please use at your own risk.!!!!!
Please modify per needs of the data/project
*/
Names Default To Here( 1 );
zProp = 0.1;
dt = Current Data Table();
w = New Window( "Select Standard Deviation Limit",
<<MODAL>= pctdone,
tb << set text( "Checking Column " || Char( i ) || " of " || Char( N Items( selcols ) ) || " selected columns" );
pctdone += pctinc;
Wait( 0 );
);
v = Column( dt, selcols[i] ) << get values;
If( std dev( v ) <= zProp,
Insert Into( collist, Column( dt, selcols[i] ) ),
Insert Into( notcollist, Column( dt, selcols[i] ) )
);
);
For( i = 1, i <= N Items( cbsellist ), i++,
Match( cbsellist[i],
1,
// group acceptable
tb << set text( "grouping acceptable columns" );
Wait( 0 );
dt << group columns( "acceptable cols", notcollist );
dt << clear column selection();
dt << select column group( "acceptable cols" );
dt << move selected columns( to last );
dt << clear column selection();
tb << set text( "" );
Wait( 0 );,
2,
// group unacceptable
tb << set text( "grouping unacceptable columns" );
Wait( 0 );
dt << group columns( "unacceptable columns", collist );
dt << clear column selection();
dt << select column group( "unacceptable columns" );
dt << move selected columns( to last );
dt << clear column selection();
tb << set text( "" );
Wait( 0 );,
3,
// hide and exclude
tb << set text( "hiding and excluding unacceptable columns" );
dt << clear column selection();
dt << Select Columns( collist );
Wait( 0 );
If(
Host is( "Mac" ),
Main Menu( "Cols:Hide/Unhide" );
Main Menu( "Cols:Exclude/Unexclude" );,
Host is( "Windows" ),
mainmenu( "Cols:Hide/Unhide" );
mainmenu( "Cols:Exclude/Unexclude" );
);
dt << clear column selection();
tb << set text( "" );
Wait( 0 );
)
);
),
Button Box( "Cancel" )
)
)
)
)
)
);</MODAL>
Re: outlier detection approach - iterative 3 sigma - Status changed to: Acknowledged
