How to compare Group Differences with small sample sizes ?

Question

Hello everyone,

I have 3 different groups and I'm aiming to compare them to see if there are any statistically significant differences. However, my sample sizes are quite small: Group 1 has 3 samples, Group 2 has 2 samples, and Group 3 has 3 samples. Due to this small sample size, I'm unable to test for normality.

To analyze the data, I conducted a Kruskal-Wallis test followed by a Steel-Dwass test (as a post-hoc analysis). I've attached a screenshot of the output for one of the variables, but I have similar results for other variables as well.

Now, I have a few questions regarding the results:

* The footnote below the Kruskal-Wallis table mentions "small sample sizes" and suggests referring to statistical tables for tests rather than relying on large-sample approximations. Could someone please clarify what this means, and what to do in that case?

* I noticed that the Kruskal-Wallis test reported significant differences in variance, while the Steel-Dwass test reported no differences at all. Is this a common outcome that a post-hoc test reports different results?

* Lastly, I would appreciate any suggestions for alternative tests or approaches to assess significant differences between the three groups. Initially, I didn't conduct any tests due to the small sample size, but a reviewer of my article requested that I implement a test to assess differences.

Thank you for your help!

Test 1.PNG

MRB3855 · Answer

Hi @ELH   The residuals should be normally distributed (residuals are the data “corrected”, via subtracting group mean, for differences between groups so you can assess normality of residuals wrt to all observations). But as @SamGardner  said, a small departure from normality is nothing to worry about. Constant variance is, IMHO, more important than normality; informally, this means the spread of data within each group should be about the same for each group. So, in general, if your residual plot (residuals vs. predicted) looks random with no pattern to speak of, then keep calm and carry on.

ELH · Answer

@SamGardner I see you did a normality test by combining all the data points so you ended up with 9 data points, I thought you were supposed to do the test for each group

SamGardner · Answer

You can still do the ANOVA, save the residuals from the ANOVA (it will save them to a column named " Centered by "model and make a histogram, QQ plot, ... This is me speaking as a statistician...don't make so much of normality. It will be ok. The only reason it might not if is you have a very large outlier or you knew that the the process that produces the results will have a response distribution that is markedly non-normal. ANOVA is fairly robust to small to moderate departures from normality.