<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />A colleague and I discovered this interesting behavior in Categorical. I think I know what is going on but it's counter-intuitive to our users here.
In the following, the comparison of interest is column B vs. E, but the full data set contains 16 total products. If we look at all 16 products together, we get this (note B vs. E has a small e, so significant at alpha = 0.10):
If we then use a Local Data Filter to grab just the two products we're interested in (B and E), we get this:
Same means, same std. devs., but now Compare Means has a capital E indicating significance at alpha = 0.05 instead of the lower case e.
We don't expect the capitalization to change. Understandably, there are multiple comparison considerations and other issues in the full 16-product view. At the same time, including the larger variances especially of our products L through P (not shown) leads to a deflated t statistic and causes a loss of significance, in part because L through P are inferior products that score lower and consequently have a higher standard deviation.
Can we have an option to turn off the pooling of standard deviation, essentially doing multiple two-sample t-tests each assuming unequal variances?
Thanks,
Michael
Yes, it does appear as JMP is using a pooled estimate of standard deviation rather than doing a series of 2-sample t-tests. I was under the impression that it was doing 2-sample t-tests (thus not doing pooled sd). The help file would lead me to believe that it was doing a series of 2 sample t-tests. I would like to better understand what is meant by multiple comparisions in help as listed below.
Help in JMP
Mean Score Comparison
Compares the mean scores across groups of sample levels, showing which groups are significantly different. A pairwise multiple comparisons Student’s t test is used for the mean score comparison, based on the specified comparison groups. For more information about the letter codes, refer to "Comparisons with Letters". For more information about specifying the comparison groups, refer to "Specify Comparison Groups".
