Automatically selecting outliersI have 7000 rows, that consists of multiple batches. I am doing a distribution by batch and using the outlier box plot to look for outliers. But I have 100+ batches, is there a way to have JMP automatically select which rows are outliers?
Re: Automatically selecting outliersis this what you need?
Names Default To Here( 1 );
// make a sample data table
dt = New Table( "Data",
add rows( 10000 ),
New Column( "it", Numeric, Ordinal, Format( "Best", 8 ), Formula( Row() ) ),
New Column( "Batch", Numeric, Ordinal, Format( "Best", 8 ), formula( Random Integer( 110 ) ) ),
New Column( "y1", Numeric, Continues, Format( "Best", 8 ), formula( Random Lognormal( :Batch / 10, 5 ) ) ),
New Column( "y2", Numeric, Continues, Format( "Best", 8 ), formula( Random Lognormal( :Batch / 10, 5 ) ) ),
New Column( "Out",
Numeric,
Ordinal,
Format( "Best", 8 ),
Set Property( "Value Colors", {0 = -13977687, 1 = -4222943} ),
Color Cell by Value,
formula( 0 )
),
New Column( "Outy1",
Numeric,
Ordinal,
Format( "Best", 8 ),
Set Property( "Value Colors", {0 = -13977687, 1 = -4222943} ),
Color Cell by Value,
formula( 0 )
),
New Column( "Outy2",
Numeric,
Ordinal,
Format( "Best", 8 ),
Set Property( "Value Colors", {0 = -13977687, 1 = -4222943} ),
Color Cell by Value,
formula( 0 )
),
);
dt << run formulas;
Column( dt, "it" ) << delete formula;
Column( dt, "Batch" ) << delete formula;
Column( dt, "y1" ) << delete formula;
Column( dt, "y2" ) << delete formula;
Column( dt, "Out" ) << delete formula;
Column( dt, "Outy1" ) << delete formula;
Column( dt, "Outy2" ) << delete formula;
// now we get to work
// make a list of batches
byvarlist = Associative Array( dt:Batch ) << get keys;
// make a list of columns
cols = dt << get column names( Continuous ); //List all Numeric columns
// loop through the batches
For( i = 1, i <= N Items( byvarlist ), i++,
// select each batch
dt << select where( dt:Batch == byvarlist[i] );
// subset each batch
subdt = dt << Subset( Selected Rows( 1 ) );
// clear selection for smoth workflow
dt << clear select;
// finde outliers in each variable within each batch
For( ii = 1, ii <= N Items( cols ), ii++,
out = Eval(
Parse(
"subdt << get rows where(
subdt:" || Char( cols[ii] ) || " > Col Quantile( subdt:" || Char( cols[ii] ) ||
", 0.75 ) + 1.5 * (Col Quantile( subdt:" || Char( cols[ii] ) || ", 0.75 ) - Col Quantile( subdt:" || Char( cols[ii] ) ||
", 0.25 )) | subdt:" || Char( cols[ii] ) || " <
Col Quantile( subdt:" || Char( cols[ii] ) ||
", 0.25 ) - 1.5 * (Col Quantile( subdt:" || Char( cols[ii] ) || ", 0.75 ) - Col Quantile( subdt:" || Char( cols[ii] ) ||
", 0.25 )));"
)
)`;
Eval( Parse( "column (subdt, \!"Out" || Char( cols[ii] || "\!" )[" || Char( out ) || "] = 1;" ) ) );
// also indicate the global outlier
Column( subdt, "Out" )[out] = 1;
//wait (0.00001);
);
// update original data table with new information for the batch
subdt << delete columns( "Batch" );
subdt << delete columns( cols );
dt << Update( With( subdt ), Match Columns( :it = :it ) );
Close( subdt, No Save );
//Wait( 0.00001 );
);
dt << color by column( :Out );
Re: Automatically selecting outliersJust one more quick question. I deleted about 90 columns, can this work
with n columns or does it always have to be 3 columns?
Re: Automatically selecting outliersSeems to work. thanks.
Re: Automatically selecting outliersnow see if it works for you...
the script you sent was not adapted to your data at all...
Re: Automatically selecting outliersattached is an example of my data I was trying to use this with. Ideally it would do this for each Y.
Re: Automatically selecting outliersmy formula is not sensitive to batch size. furthermore, it will work even if batch column is character (and not numeric).
can you post a sample, even with just 2 or three different batches?Re: Automatically selecting outliersWhen I run it, I get one table, and here is a snapshot of it, this will
work since I can select rows where Out = 1, if I want to select all of the
outliers.
[image: image.png]
Re: Automatically selecting outliersif you run the whole script can you see that you get what you wanted (just not on your data)?Re: Automatically selecting outliersI copy everything from //now we get to work on down. When I run this in
my own table, it just creates one table with one batch only. I must say
one thing, is that the number of points per batch are not consistent. I do
not know if that matters.
Re: Automatically selecting outliersthis worked for me.
at the end you should have 3 tables. original with all rows, one with just outliers and the last without outliers. let me know if it works.
Names Default To Here( 1 );<BR /><BR />// make a sample data table<BR />dt = New Table( "Data",<BR /> add rows( 10000 ),<BR /> New Column( "it", Numeric, Continues, Format( "Best", 8 ), Formula( Row() ) ),<BR /> New Column( "Batch", Numeric, Ordinal, Format( "Best", 8 ), formula( Random Integer( 110 ) ) ),<BR /> New Column( "y", Numeric, Continues, Format( "Best", 8 ), formula( Random Lognormal( :Batch / 10, 5 ) ) ),<BR /> New Column( "Out", Numeric, Ordinal, Format( "Best", 8 ), formula( 0 ) ), <BR /><BR />);<BR /><BR />dt << run formulas;<BR />Column( dt, "it" ) << delete formula;<BR />Column( dt, "Batch" ) << delete formula;<BR />Column( dt, "y" ) << delete formula;<BR />Column( dt, "Out" ) << delete formula;<BR /><BR />// now we get to work<BR />dt = Current Data Table();<BR />byvarlist = Associative Array( dt:Batch ) << get keys;<BR />For( i = 1, i <= N Items( byvarlist ), i++,<BR /> dt << select where( dt:Batch == byvarlist[i] );<BR /> subdt = dt << Subset( Selected Rows( 1 ) );<BR /> dt << clear select;<BR /> subdt:Out << set formula(<BR /> If(<BR /> :y > Col Quantile( :y, 0.75 ) + (Col Quantile( :y, 0.75 ) - Col Quantile( :y, 0.25 )) * 1.5, 1,<BR /> :y < Col Quantile( :y, 0.25 ) - (Col Quantile( :y, 0.75 ) - Col Quantile( :y, 0.25 )) * 1.5, 1,<BR /> 0<BR /> )<BR /> );<BR /> subdt << run formulas;<BR /> Column( subdt, "Out" ) << delete formula;<BR /> keep = {"it", "Out"};<BR /> todelete = subdt << get column names( "string" );<BR /> For( icol = 1, icol <= N Items( keep ), icol++,<BR /> Remove From( todelete, Contains( todelete, Keep[icol] ), 1 )<BR /> );<BR /> subdt << delete columns( todelete );<BR /> dt << Update( With( subdt ), Match Columns( :it = :it ) );<BR /> Close( subdt, No Save );<BR /> Wait( 0.00001 );<BR />);<BR />selected = dt << select where( :Out == 1 );<BR />selected << Markers( 11 );<BR />selected << Marker Size( 8 );<BR />selected << Row Colors( red );<BR />dt << clear select;<BR /><BR />dt << select where( dt:Out == 1 );<BR />Outdt = dt << Subset( Selected Rows( 1 ) );<BR />dt << clear select;<BR /> <BR />Wait( 0.00001 );<BR /> <BR />dt << select where( dt:Out == 0 );<BR />indt = dt << Subset( Selected Rows( 1 ) );<BR />dt << clear select;<BR /><BR />
Re: Automatically selecting outliersI am not sure if you got my last message I just sent, I got a weird
automated email back. I fixed the second one and it is still doing the
same thing.
Re: Automatically selecting outliersin the for loop it isn'tRe: Automatically selecting outliersMy batch column is capitilzed:
[image: image.png]
Re: Automatically selecting outliersthe column name is case sensitive, be consistent when using :Batch or :batchRe: Automatically selecting outlierswhen I ran this, it made a subset table with the first batch only. I
changed this to do my open table:
dt = currentdatatable();
byvarlist = Associative Array( dt:Batch ) << get keys;
For( i = 1, i <= N Items( byvarlist ), i++,
dt << select where( dt:batch == byvarlist[i] );
subdt = dt << Subset( Selected Rows( 1 ) );
dt << clear select;
subdt:Out << set formula(
If(
:y > Col Quantile( :y, 0.75 ) + (Col Quantile( :y, 0.75 )
-Col Quantile( :y, 0.25 )) * 1.5, 1,
:y < Col Quantile( :y, 0.25 ) - (Col Quantile( :y, 0.75 )
-Col Quantile( :y, 0.25 )) * 1.5, 1,
0
)
);
subdt << run formulas;
Column( subdt, "Out" ) << delete formula;
keep = {"it", "Out"};
todelete = subdt << get column names( "string" );
For( icol = 1, icol <= N Items( keep ), icol++,
Remove From( todelete, Contains( todelete, Keep[icol] ), 1 )
);
subdt << delete columns( todelete );
dt << Update( With( subdt ), Match Columns( :it = :it ) );
Close( subdt, No Save );
);
selected = dt << select where( :Out == 1 );
selected << Markers( 11 );
selected << Marker Size( 8 );
selected << Row Colors( red );
dt << clear select;Re: Automatically selecting outlierslet me try again
it is not a sleek solution but steps are clear and easy to modify
Names Default To Here( 1 );
// make a sample data table
dt = New Table( "Data",
add rows( 10000 ),
New Column( "it", Numeric, Continues, Format( "Best", 8 ), Formula( Row() ) ),
New Column( "Batch" ,Numeric, Ordinal, Format( "Best", 8 ), formula( random integer (110) )),
New Column( "y" ,Numeric, Continues, Format( "Best", 8 ), formula( random log normal (:Batch / 10, 5) )),
New Column( "Out" ,Numeric, Ordinal, Format( "Best", 8 ), formula (0)),
);
dt << run formulas;
column (dt, "it") << delete formula;
column (dt, "Batch") << delete formula;
column (dt, "y") << delete formula;
column (dt, "Out") << delete formula;
// now we get to work
// extract the list of the by variable levels
byvarlist = Associative Array( dt:Batch ) << get keys;
// run across each group and locate the outliers
For( i = 1, i <= N Items( byvarlist ), i++,
// produce a subset data table for each group using selection method
dt << select where (dt:batch == byvarlist[i]);
subdt = dt << Subset(
Selected Rows( 1 ),
);
dt << clear select;
// indicate outliers within the group
subdt:Out << set formula(
If(
:y > Col Quantile( :y, 0.75 ) + (Col Quantile( :y, 0.75 ) -
Col Quantile( :y, 0.25 )) * 1.5, 1,
:y < Col Quantile( :y, 0.25 ) - (Col Quantile( :y, 0.75 ) -
Col Quantile( :y, 0.25 )) * 1.5, 1,
0
)
);
subdt << run formulas;
// prepare table for updating original by removing formulas and columns that are not needed
column (subdt, "Out") << delete formula;
keep = {"it", "Out"};
todelete = subdt << get column names( "string" );
For( icol = 1, icol <= N Items( keep ), icol++,
Remove From( todelete, Contains( todelete, Keep[icol] ), 1 )
);
subdt << delete columns( todelete );
// update
dt << Update(
With( subdt ),
Match Columns( :it = :it )
);
Close( subdt, No Save );
);
// now the original data has all the outliers indicated in a column
selected = dt << select where (:Out == 1);
selected << Markers( 11 );
selected << Marker size ( 8 );// not working???
selected << Row Colors( red );// not working???
dt << clear select;
Re: Automatically selecting outlierstry to split the table by batch and group by unique value (so you do not lose any rows) then run my script.
otherwise you need a different strategy than the one i offer.
does it make sense?Re: Automatically selecting outliersI can't get this to work with my data, I changed the height and weight to my columns and removed the two lines where it opened the table and embedded it into my table, but it says evaluations are done but will not do anything.
I also need something that uses the by and this for each batch individually.
Re: Automatically selecting outliersHi,
perhaps here is a solution:
https://community.jmp.com/t5/Discussions/How-to-extract-month-in-words-from-a-date-value-how-to-change/m-p/19362/highlight/true#M17650
Ron
Re: Automatically selecting outliersI tried the Grubb's script. This just shows the outliers and does not highlight the rows right? I need something that will highlight the rows.
Re: Automatically selecting outliersThe screening outliers does not work if I need a variable in the by. What I want to do is my Y is in one column. Another column is labeled batch. I put batch in by and make the number of distributions as I do batches. So say 100 distributions. I want a script to highlight all the outliers in all 100 distributions. Ideally it will only highlight the rows so I can pick my own color since I use colors for different thing. Make more sense now?
Re: Automatically selecting outliersIf you had searched the File Exchange portion of the JMP Community for 'outlier,' you would have found these two utility scripts:
Of course, the statistical basis for each of these tests must be suited to your situation and criterion.
Re: Automatically selecting outliersI am not sure of exactly what you are looking for. If you want to select the rows for a given distribution, when viewing the output, you can simply drag across the displayed outlier data points in the outlier box plot, and it will select the outlier data points.
But if you want to go from column to column and select outlier rows, a simple script could do that, however, remember that row selection goes across all columns, and what you would end up will are selected rows where any of the columns are selected.
What you may really want, is a simple script that colors all of the individual cells that are outliers. Maybe red if they are to far above the mean, and orange if they are outliers below the mean.
Can you please clarify exactly what you are looking for?
See Dan's reply for a better answer.
Re: Automatically selecting outliersYou might want to look under Analyze > Screening > Explore Outliers.
