outlier detection approach - iterative 3 sigma

flo_kussener

What inspired this Wish List Request? 

The aim is to filter the data and remove outliers in order to set aside wafers with many defects and identify the standard. the data follows a poisson distribution

Please describe the new or improved feature or solution you would like to see.  

they would like to detect outlier by using iterative 3 sigmas approach. 

it means:

1. Measure mean and standard deviation of samples.
2. Remove samples higher than mean plus 3 standard deviations.
3. Repeat from Step 1 - for a total of 3 times.

What would the value be to you and/or other users if this new feature is provided? 

 

Is this an external customer request? 

STMicroelectronics - Crolles

Is there a competitive software package that has this feature or solution that we should be aware of? 

Don't know

Will this impact a renewal, expansion, or adoption at a customer, and does this put an account at risk if we do not address this? 

On-going ELA at STMicroelectronics

Chris_Kirchberg

Hi @flo_kussener, 

Would something like the below be something that could be adapted to the customer needs in the mean time? It is only a single pass for a defined standard deviation using std dev() function in JSL.

/*
This script was created by:
Chris Kirchberg, Principal Systems Engineer, JMP Statistical Discovery, LLC
Sam Garnder, Sr. Product Manager, JMP Statistical Discovery, LLC

Date: 2023-10-24

It is intended to show how one can filter/group/screen through thousands of columns in a wide data set to 
help reduce a data set to useful or functional features/columns for further modeling/testing.

!!!!!!It was not evaluated by any QC/QA methods or departments. Please use at your own risk.!!!!!

Please modify per needs of the data/project
*/

Names Default To Here( 1 );
zProp = 0.1;
dt = Current Data Table();
w = New Window( "Select Standard Deviation Limit",
	<<Modal,
	H List Box(
		Text Box(
			"This tool is for checking the standard deviation in each of the selected columns.
			You can choose to group columns with too many or acceptable standard deviation
			and to hide and exclude columns with unacceptable standard deviation.",
			<<set wrap( 550 )
		)
	),
	H List Box(
		V List Box(
			H List Box(
				Panel Box( "Select Columns", clb = Filter Col Selector( dt, all ) ),
				Panel Box( "Selected Expression Columns to be Evaluated",
					Lineup Box( N Col( 2 ), Spacing( 5 ),
						Button Box( "Add", cl << Append( clb << Get Selected ) ),
						cl = Col List Box( "Numeric", MinItems( 1 ) ),
						Button Box( "Remove", cl << Remove Selected )
					)
				)
			), 
		
			Border Box( top( 0 ), bottom( 0 ), Left( 0 ), Right( 20 ),
				vb = V List Box(
					Spacer Box( size( 0, 20 ) ),
					H List Box( Text Box( "Acceptable Standard Deviation in Selected Columns: " ), neb = Number Edit Box( 0.1 ) ),
					cb = Check Box(
						{"Group Acceptable Columns", "Group Unacceptable Columns", "Hide and Exclude Unacceptable Columns"}
					),
					Spacer Box( size( 1, 30 ) ),
					H List Box(
						Button Box( "OK",
							zProp = neb << get;
							rNum = N Rows( dt );
							cbsellist = cb << get selected indices;
							selcols = cl << Get Items( "Column Reference" );
							collist = List();
							notcollist = List();
							vb << Append( Spacer Box( size( 1, 30 ) ) );
							vb << Append( tb = Text Box( "In Progress" ) );
							pctinc = 2;
							pctdone = pctinc;
							For( i = 1, i <= N Items( selcols ), i++,
								If( i / N Items( selcols ) * 100 >= pctdone,
									tb << set text( "Checking Column " || Char( i ) || " of " || Char( N Items( selcols ) ) || " selected columns" );
									pctdone += pctinc;
									Wait( 0 );
								);
								v = Column( dt, selcols[i] ) << get values;
								If( std dev( v ) <= zProp,
									Insert Into( collist, Column( dt, selcols[i] ) ),
									Insert Into( notcollist, Column( dt, selcols[i] ) )
								);
							);
							For( i = 1, i <= N Items( cbsellist ), i++,
								Match( cbsellist[i],
									1, 
									// group acceptable
										tb << set text( "grouping acceptable columns" );
										Wait( 0 );
										dt << group columns( "acceptable cols", notcollist );
										dt << clear column selection();
										dt << select column group( "acceptable cols" );
										dt << move selected columns( to last );
										dt << clear column selection();
										tb << set text( "" );
										Wait( 0 );,
									2, 
									// group unacceptable
										tb << set text( "grouping unacceptable columns" );
										Wait( 0 );
										dt << group columns( "unacceptable columns", collist );
										dt << clear column selection();
										dt << select column group( "unacceptable columns" );
										dt << move selected columns( to last );
										dt << clear column selection();
										tb << set text( "" );
										Wait( 0 );,
									3, 
									// hide and exclude
										tb << set text( "hiding and excluding unacceptable columns" );
										dt << clear column selection();
										dt << Select Columns( collist );
										Wait( 0 );
										If(
											Host is( "Mac" ),
												Main Menu( "Cols:Hide/Unhide" );
												Main Menu( "Cols:Exclude/Unexclude" );, 
											
											Host is( "Windows" ),
												mainmenu( "Cols:Hide/Unhide" );
												mainmenu( "Cols:Exclude/Unexclude" );
										);
										dt << clear column selection();
										tb << set text( "" );
										Wait( 0 );
								)
							);
						),
						Button Box( "Cancel" )
					)
				)
			)
		)
	)
);