Shapiro-Wilk test for samples larger than 2000?The JMP Help page states "For Normal distributions, the Shapiro-Wilk test for normality is also reported when the sample size is less than 2000 and there are no fixed parameters."
I've got a customer that has samples larger than 2000 at times, but his internal procedures tell him to use the Shapiro-Wilk test. Several online sources mention 5000 observations as the upper limit on sample size for this test. My customer has asked why JMP limits the availability of the Shaipiro-Wilk test to 2000 observations. Would it take too long to calculate? Or is there some other reason?
Re: Shapiro-Wilk test for samples larger than 2000?If I recall correctly, the p-values need to be generated via simulation or approximation. Could be that we've only generated values up to N=2000.
Re: Shapiro-Wilk test for samples larger than 2000?Only the developer would know. As @DonMcCormack mentions the p-value is simulated for this test. SAS also limits the sample size <=2000; for n>2000 the test is not available; See the details PROC UNIVARIATE: Goodness-of-Fit Tests (sas.com)
Re: Shapiro-Wilk test for samples larger than 2000?Here is a nice reference that describes and compares several different types of normality tests (https://www.tandfonline.com/doi/pdf/10.1080/00949655.2010.520163). We briefly had removed the SW test (i think in v14 or 15.0), but brought it back in v15.1 for sample sizes less that 2000. My question is why are they testing for normality with such a large sample size? For Sample statistics, the Central Theorem will work for them, and for analysis where the actual distribution matters (estimating tolerance intervals, percentiles, control charting, perhaps?). then emperical statistics should be fine.
Re: Shapiro-Wilk test for samples larger than 2000?Thank you @SamGardner I did mention to him that with such a large sample size the test would probably be significant with even the smallest departure from normality. He realized that and agreed, and I didn't push any more. He's a statistician at Abbott Laboratories. I'll use the info you provided to follow up with him. Thank you @DonMcCormack and @jiancao for responding, too.
Re: Shapiro-Wilk test for samples larger than 2000?The wikipedia article on this test gives more detail and background. The reason it is limited to 2000 is because the test statistic uses coefficients that are based on expected value of and the inverse of the covariance martrix of standard normal distribution order statistics. This is computationally tedious, and when the method was first proposed it was not feasible to do this for large sample sizes. The SW was originally proposed for sample sizes up to 50. Royston in late 80s/early 90s came up with an approximate way to calculate those coefficients or samples size n=3 to n=1999. I haven't been able to find any other research that has extended the SW test to larger sample sizes.
Re: Shapiro-Wilk test for samples larger than 2000?I'm digging into this a bit more for another customer and wanted to take the opportunity to consolidate my thinking and thank you guys for putting this up on the community. To reiterate, Per the SAS doc, a normalizing transformation is used on the [Shapiro-Wilk] W test statistic to determine the p-values for the test (for a sample size, n ≥ 4). The constants used in that normalizing transformation are derived from simulation results which depend on n. But the p-values themselves are not simulated for the test. The constants are there for up to an n of 1,999; so, the test won't compute in JMP or SAS for n ≥ 2000. I think we could have gone up to 5000 and my guess is that it's just historical precedent (which may or may not have come from a publication reference). Why cap it at a 'large' sample size though? (aside from the computational constraints at the time the method was developed, as @SamGardner mentioned)
For S-W, it's not related to calculation time (although for A-D, it is). I believe it's related to the fact that W is computed using a covariance calculation (on the matrix of normal order statistics as @SamGardner mentioned). Covariance increases with increasing n (or dof) and that can bias the estimate of the test statistic (by inflating its standard error). 2000 might have been an arbitrary choice, but the idea in capping it is that we want to minimize that covariance inflation risk, so the W test statistic (which is a function of n) is an unbiased estimator (over thousands of samples of size n). That's my take on it.
Agree with the cautions on doing Normality testing on such a large sample size. Of course, it depends on the application, but in my med-device product mfg. experience, I've never seen sample sizes this large come from a continuous process (i.e. one with just one source of variability).
(Slides courtesy of JMP Education)
Usually it looks like this instead, where you can replace the word "Batch" with Lot (and where Lot sizes were typically on the order of several hundred).
Re: Shapiro-Wilk test for samples larger than 2000?The calculation time constraint is on the time it takes to actually use simulation or numerical integration and matrix operations to determine the S-W coefficients. I tried a simulation approach and the plot of computation time vs n is shown here. Also memory requirements increase, at least order n^2, so it just gets infeasible to do this. Also, for each sample size n you need to store a list of n coefficients to be used, so the total storage needed grows n*(n+1)/2 which is also order n^2.
But @PatrickGiuliano is correct that with modern computing, this could probably be extended to sample sizes greater than 2000, but not dramatically so.
