nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

A script Normality test for many columns

Hi,

I like to write a script that can check the normality of many columns and create a new column with binary output (0,1) to indicate normal and not normal. I have seen an addin for normality test but I don't know how I can use that to accomplish what I want.

Any help is greatly appreciated.

Thanks.

Accepted answers

All comments

David_Burnham

Not sure what your knowledge of JSL is like but you can use a user-defined function such as this:

Test Normal = Function({dt,col},{Default Local},
	
	dist = dt << Distribution( Invisible,
		Continuous Distribution(
			Column( Eval(col) ),
			Fit Distribution(
				Normal( Goodness of Fit( 1 ) )
			)
		),
		Histograms Only
	);
	rep = dist << Report;
	ob = rep["Goodness-of-Fit Test"];
	tb = ob[TableBox(1)];
	mat = tb[NumberColBox(2)] << Get;
	pValue = mat[1];
	rep << Close Window;
	Return(pValue);
	
);

// illustration of using hte function

dt = Open("$SAMPLE_DATA/Big Class.jmp");
pValue = Test Normal(dt,"height");
show(pValue);
If (pValue<0.05,
	show("failed normality test")
,
	show("passed normality test")
);

From here, you can put the function inside a for-loop that iterates over the columns:

lstNames = dt << Get Column Names(String,Numeric);
For (i=1,i<=NItems(lstNames),i++,
    pValue = Test Normal(dt,lstNames[i]);
   // etc
);

David_Burnham

Here is a discussion about the workflow associated with constructing the user-defined function

http://www.pega-analytics.co.uk/blog/testing-residuals-for-normality/

Thanks so much for quick feedback and solution. I used the template you sent me and when I run the last loop:

For (i=1,i<=NItems(lstNames),i++,
    pValue = Test Normal(dt,lstNames[i]);
    show(pValue);
If (pValue<0.05,
show("failed normality test")
,
show("passed normality test"))
);

I get an error message:

Cannot subscript Display Box in access or evaluation of 'Subscript' , rep[/*###*/"Goodness-of-Fit Test"]

I appreciate your help. Thanks.

);

David_Burnham

That loop was just for illustration. With the Big Class data it doesn't make much sense. The error is because it is trying to test normality for the age column - which has an ordinal modelling type - whereas the test for normality assumes the column is continuous. So you just need to provide it with some meaningful input data.

David_Burnham

Some better data ... for example

dt = Open("$SAMPLE_DATA/Semiconductor Capability.jmp");

For (i=5,i<=10,i++,

&nbsp; &nbsp; colName = Column(dt,i) << Get Name;

&nbsp; &nbsp; show(colName);

&nbsp; &nbsp; pValue = Test Normal(dt,colName);

&nbsp; &nbsp; show(pValue);

);

Hi David,

Thanks. I tried with a different table and looked at one column. I got the result that the distribution is not normal and I am surprised because the distribition is normal to me. The reported p value is 0.01. I have attached the histogram. Any comments? Thanks.

Histogram.png