Can I exclude an apparent outlier while doing multiple comparisons?The attached JMP file shows a multiple comparison situation where the exclusion of an apparent outlier in row 60 leads to different results.
What would be the best practice here?
When row 60 is included in the Dunnett’s test, the mean for the control R1 equals those of R3 and R5 (in red).
When row 60 is excluded, the mean for the control R1 equals those of R3, R4, and R5.
Re: Can I exclude an apparent outlier while doing multiple comparisons?Others with more statistical savvy than I should weigh in but the first question I would ask before excluding an outlier is whether you know of an assignable cause for that point? That is, do you know what caused it to have that value or was it normal variation?
If you don't know a specific cause then you probably have to assume it's normal variation and you probably shouldn't exclude it just because it looks like an outlier AND it affects your analysis.
Re: Can I exclude an apparent outlier while doing multiple comparisons?The fact that by including the outlier you get the R4-R1 difference with p<0.05 (significant) and by excluding it with p-value>0.05 does not seem to make a practical difference. It is just that we tend to be too concerned with whether the p-value < 0.05. There has been a lot of discussions these days among statisticians on this binary way of thinking (significant or non-significant) which leads to confusion as the one you are having here.
A more modern approach is to look at the confidence intervals (CI). Instead of using “Fit X by Y” to have the comparisons, you can do the same thing by Analyze>Fit Model, select Y = Y, and X = X, and then hit “Run”. In the model report window that pops up, click on the red triangle and select Estimate>Multiple Comparisons. Select the Dunnett’s test and R1 as control. This will generate the “Multiple Comparisons for X” tab in your report, only that now you have the CIs which is not possible to get via “Fit X by Y” (maybe we should put it in our JMP wish-list). The two pictures below show this report while including and then excluding row 60. The script to generate this test is saved in the attached JMP table.
By including row 60, the CI of R4-R1 is 0.10297 to 6.4614; by excluding it the CI is -0.29514 to 5.8452. Whether the lower CI is 0.10297 or -0.29514 seems irrelevant from a practical stand point because there are both very close to zero. So, the fact significance is lost when you exclude row 60 does not mean a big deal.
As such, I’d retain row 60 and also report CIs. Not only because of the above point but also because excluding outliers, and the method to do so, is something that should be agreed upon before you know the data. If done after the fact it can lead to data dredging, unless you know with almost certainty why row 60 was that low (an assignable cause as @Jeff_Perkinson just mentioned).
