I’d like to know how the choice of the number of samples per group impacts these two tests as a function of the number of groups I want to test.
This topic stems from power analysis. Small, unimportant in practical terms, differences between group means can always be observed as significant provided sample size is big enough. Conversely, larger important mean differences can be observed as non-significant if sample size is not big enough. Efficient group sample size (not too big, not too small) is defined as a function of (a) the desired probability (i.e. power, typically 80%) of observing the differences between group means as significant if they are truly not zero, (b) the size of the critical (i.e. smallest) difference to be detected, (c) the desired level of confidence (typically 95%), (d) the within group std deviation, and (e) the number of groups to be compared. As such, in order to calculate an efficient group sample size it is necessary to have all these five factors defined beforehand. The critical difference and the within group std deviation are usually not known a priori, mainly in R&D.
One solution to this riddle starts by thinking about the critical difference to be detected in terms of number of within group std deviations, which is known as “effect size” (ES). Thus, ES is the ratio of the critical difference to the within group std deviation. The plot below shows the required group size (n/group) as a function of (i) ES, (ii) number of groups means being tested, and (iii) type of test — all assuming power = 80%, alpha = 0.05 (95% confidence), normality, independent errors, equal variance across groups, and equal group size.
This plot shows the difference in the required n/group for a given ES level between the Tukey’s and Dunnett’s test. When the number of groups equals two, both tests replicate a t-test and require the same n/group. As the number of groups grow or as ES gets smaller, the Dunnett’s test requires increasingly lower n/group. This is because under Dunnett’s, only a control group is compared to all other groups; under Tukey’s, all groups are compared to all groups. The higher amount of comparisons under Tukey’s demands a larger n/group.
Thus, whenever the experimental objective can be satisfied by a comparison of all groups to a control group, be it the competition’s solution or any other type of incumbent solution, the Dunnett’s test should be used as it requires lower, hence cheaper, n/group.
In absence of a priori knowledge of ES, our experience tells that ES=1 is a very plausible worst case scenario to be assumed. Thus, according to the information of this plot, for ES=1 and:
(a) Number of groups=2, the required n/group=17 regardless the test, so the total number of samples is 34 under both tests;
(b) Number of groups=8, the required n/group=25 under Dunnett’s vs. 31 under Tukey’s, and total sample size=200 under Dunnett’s vs. 248 under Tukey’s;
(c) Number of groups=32, the required n/group=30 under Dunnett’s vs. 43 under Tukey’s, and total sample size= 960 under Dunnett’s vs. 1376 under Tukey’s.
The attached JMP table provides all the above calculations and the plot script itself.
