49250_en-US

liliconcom

A script Normality test for many columns

Hi,

I like to write a script that can check the normality of many columns and create a new column with binary output (0,1) to indicate normal and not normal. I have seen an addin for normality test but I don't know how I can use that to accomplish what I want.

 

Any help is greatly appreciated.

 

Thanks.

Re: A script Normality test for many columns

Thank you very much.

Re: A script Normality test for many columns

Jim's idea of random sampling is a good one but then you must face the issue of sample size and power. It might be more straight-forward to adjust your p-values using the false detection rate (FDR) approach.

Re: A script Normality test for many columns

You could radomly sample from your data down to a reasonable sample size so you don't drown the statistics with sample size.

Re: A script Normality test for many columns

Thanks Dan for the comments. But what should I do then. I am trying to understand if the distributions are normal or not and I do have large samples.

 

I appreciate your help on this. Thanks

Re: A script Normality test for many columns

I agree that the data look normal. You have encountered the problem with statistical testing with large amounts of data. Any difference, no matter how slight, will show as statistically significant. You have over 100,000 observations. The test for normality (like any statistical test) will find a statistically significant difference due to the large sample size. That statistically significant difference is not likely to be a practically significant difference.

Re: A script Normality test for many columns

Hi David,

 

Thanks. I tried with a different table and looked at one column. I got the result that the distribution is not normal and I am surprised because the distribition is normal to me. The reported p value is 0.01. I have attached the histogram. Any comments? Thanks.

Re: A script Normality test for many columns

Some better data ... for example

dt = Open("$SAMPLE_DATA/Semiconductor Capability.jmp");

For (i=5,i<=10,i++,

&nbsp; &nbsp; colName = Column(dt,i) << Get Name;

&nbsp; &nbsp; show(colName);

&nbsp; &nbsp; pValue = Test Normal(dt,colName);

&nbsp; &nbsp; show(pValue);

);

 

Re: A script Normality test for many columns

That loop was just for illustration.  With the Big Class data it doesn't make much sense.  The error is because it is trying to test normality for the age column - which has an ordinal modelling type - whereas the test for normality assumes the column is continuous.  So you just need to provide it with some meaningful input data.

Re: A script Normality test for many columns

Thanks so much for quick feedback and solution. I used the template you sent me and when I run the last loop:

 

For (i=1,i<=NItems(lstNames),i++,
    pValue = Test Normal(dt,lstNames[i]);
    show(pValue);
If (pValue<0.05,
show("failed normality test")
,
show("passed normality test"))
);

 

I get an error message:

 

Cannot subscript Display Box in access or evaluation of 'Subscript' , rep[/*###*/"Goodness-of-Fit Test"]

 

I appreciate your help. Thanks.

 

);

Re: A script Normality test for many columns

Here is a discussion about the workflow associated with constructing the user-defined function 

http://www.pega-analytics.co.uk/blog/testing-residuals-for-normality/

 

Re: A script Normality test for many columns

Not sure what your knowledge of JSL is like but you can use a user-defined function such as this:

Test Normal = Function({dt,col},{Default Local},
 
 dist = dt << Distribution( Invisible,
  Continuous Distribution(
   Column( Eval(col) ),
   Fit Distribution(
    Normal( Goodness of Fit( 1 ) )
   )
  ),
  Histograms Only
 );
 rep = dist << Report;
 ob = rep["Goodness-of-Fit Test"];
 tb = ob[TableBox(1)];
 mat = tb[NumberColBox(2)] << Get;
 pValue = mat[1];
 rep << Close Window;
 Return(pValue);
 
);

// illustration of using hte function

dt = Open("$SAMPLE_DATA/Big Class.jmp");
pValue = Test Normal(dt,"height");
show(pValue);
If (pValue<0.05,
 show("failed normality test")
,
 show("passed normality test")
);

From here, you can put the function inside a for-loop that iterates over the columns:

lstNames = dt << Get Column Names(String,Numeric);
For (i=1,i<=NItems(lstNames),i++,
    pValue = Test Normal(dt,lstNames[i]);
   // etc
);