nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Simple and honest explanation for two-sample t test?

yusuke_ono

This is just an informal talk, not related to work.

I have worked around statistics for about 30 years, I still cannot find a good way to explain even the two-sample t-test, a very simple and popular statistical test.

I would like to explain the assumptions as well as the (slightly limited) robustness. The following is my poor try. Are there any good textbooks to explain these things simply and honestly?

Even if 3 + 5 = 8, the solution for x1 + x2 = 8 is not only x1 = 3 and x2 = 5. The same thing often happens also in statistics.

The t-test was originally derived from X_1, ..., X_n1 -- i.i.d. N(mu_1, sigma^2) and Y_1, ..., Y_n2 -- i.i.d. N(mu_2, sigma^2), but it does not mean that the t test statistic is distributed as t-distribution only when the data is generated from this specfic model.

But the robustness changes for each situation.

(1) The violation for i.i.d. causes tragic results. For example, if you collect political opinions from Twitter (X), and try to infer to general population in Japan, the result becomes highly biased.

(2) Even when the assumption of equal variances is violated, the result is still approximately valid if sample sizes of two groups are almost equal.

(3) Even when the assumption of normality is violated, the result is still approximately valid if sample size is relatively large and if the distribution is not so skewed or so heavily tailed.

(4) The two-sample t test is also the approximation to a randomization test if sample size is relatively large. So, you can think the two-sample t test as a nonparametric test with the null hypothesis is H0: F1(y) = F2(y) for all y (although many cook books say that the t-test is a parametric test).

Accepted answers

All comments

Mark_Zwald

The statistics knowledge portal has a nice explanation of it: https://www.jmp.com/en_us/statistics-knowledge-portal/t-test/two-sample-t-test.html

yusuke_ono

Thank you for information, @Mark_Zwald -san.

I pesonally think the assumptions and robustness of (Student's and Welch's) two-sample t-test are more comples.

As an example, the following explanation is written in the page.
>What if the variances for my two groups are not equal?
>You can still use the two-sample t-test. You use a different estimate of the standard deviation.

I guess the above sentences suggest that if variances are equal, you should use Student's t-test, and if variances are unequal, you should use Welch's t-test. (The "standard deviation" means "standard error" I guess.) But if the sample sizes are roughly equal, the results of Student's t-test and the Welch's t-test are very close. In addition, if the variances of two groups are very close, again, the results of Student's t-test and Welch's t-test are very close.

The current above explanation seems to say that we cannot use Welch's t test for equal variances, or we always cannot use Student's t test for unequal variances.

The other property I think important is that the two-sample t-test is an approximation to a randomization test. This property gives the validness to the use of t-test for randomzied clinical trials. The clinical trials never use random sampling. It uses only the random allocation.So, their data is not random samples. Even so, the two-sample Student's t-test is valid as a randomization test for H0:F1(x) = F2(x).

Chandramouli_R

Hi @yusuke_ono

Untitled design (3).png

The two-sample t-test is a widely used statistical method designed to determine whether there is a significant difference between the means of two independent groups. It is often applied in research scenarios where one seeks to compare the average outcomes of two distinct groups to assess whether any observed differences are statistically significant or could have occurred by chance.

Aspect	Description	Example
Assumptions of the Two-Sample t-Test
Independence of Observations	The observations in each group must be independent of each other.	Studying the effect of a new teaching method by testing students who study together in groups. If students discuss and help each other during the test, their scores are no longer independent
Normality	The data in each group should be approximately normally distributed, especially important with small sample sizes.	Measuring income in two cities where income distribution is skewed.
Equal Variances	The variances (spread or variability) of the two groups should be approximately equal.	Comparing consistent data in one group with highly variable data in another group.
Robustness of the Two-Sample t-Test
Violations of Independence	The t-test is not robust to violations of independence, leading to biased results.	Surveying students in a class who discuss their answers - as described above (leads to biased results).
Violations of Normality	The t-test is fairly robust to violations of normality, especially with large sample sizes.	Comparing reaction times for a video game for two groups with slightly skewed data.
Violations of Equal Variances	The t-test is moderately robust to violations of equal variances, especially if sample sizes are similar. Use Welch’s t-test when variances are unequal and sample sizes differ.	Comparing test scores from two schools with different score variances.

Please Note:

While Welch’s t-test offers a more flexible approach by accommodating unequal variances, in many practical scenarios where sample sizes are equal or variances are similar, the results from both Student’s t-test and Welch’s t-test will converge closely.
While the two-sample t-test is classified as a parametric test, assuming that the data comes from normally distributed populations. However, when sample sizes are sufficiently large, the t-test can also be viewed as an approximation to a nonparametric test. In this context, it tests the null hypothesis that the two groups’ distributions are identical, making it somewhat analogous to a nonparametric test under certain conditions.

On your last point on Clinical Trials: //In Clinical trialsTwo-Sample t-Test as an Approximation to a Randomization Test//

Given design of randomly allocating subjects to a treatment arm, the data collected in clinical trials are not random samples in the traditional sense.

However, the two-sample t-test remains valid in this context due to its ability to approximate a randomization test in a physiological and pharmacological sense (on top of rigid inclusion criteria) - means that the t-test can still accurately test the null hypothesis the two-sample t-test provides a reliable method for hypothesis testing in randomized trials, despite the absence of random sampling.

This refers to the idea that, under the null hypothesis, the distribution of the t-test statistic (in the context of randomized experiments) approximates the distribution of the test statistic obtained from a permutation or randomization test.This property justifies the use of the two-sample t-test in randomized trials, where random allocation rather than random sampling is used, because the t-test can approximate the results of a full randomization test, making it valid even in the absence of traditional random sampling.

Statistically it is refered to as randomization-based inference.

Popular references:

An Introduction to Statistical Learning

Practical Statistics for Data Scientists

Statistics: The Art and Science of Learning from Data, Third Edition (recomended)

Primary References:

Statistical Methods

Clincal Trial References:

Randomization Tests

Design and Analysis of Clinical Trials: Concepts and Methodologies Refer to Chapters (3,4 & 5)

Trust these inputs helps you.

russ_wolfinger

@yusuke_ono It's good and fun to deeply question what is arguably the most fundamental and practically useful statistical test we have. Looks like you have the beginnings of a white paper, something like "Considerations on the Remarkable Two-Sample T-Test". The natural starting point is the original test with all assumptions fully satisfied. Then consider various directions and degrees of departure from these assumptions and their impact on the sampling properties of the test (size and power).

From an engineering viewpoint, the t-test statistic is very intuitively a signal-to-noise ratio.

The connections with randomized inference are compelling and under-appreciated.

@Chandramouli_R That's some great summary material. I'd like to see mixed models added to the flow chart for the non-independent branch, the most common case being pairing or, more generally, blocking. We have excellent mixed model theory with Kenward-Roger degrees of freedom computations that still use approximate t-distributions for testing and have well-performing small-sample properties.

yusuke_ono

Thank you for your idea, @Chandramouli_R -san.

I like the explanation in the text part. Although this is just my personal preference, I personally do not like the flow chart itself at the top (although I see it in many cookbooks).

At first, the Wilcoxon's test does not maintain the alpha error when two-variances are not equal (even if the sample sizes of two groups are equal). This is called "Nonparametric Behrens-Fisher problem". There is Welch's version of Wilcoxon test (Brunner,E. and Munzel,U. (2000) The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample Approximation.)

Secondly, although I also often hear the urban legend like that "the central limit theory doesn't hold for n<=30", but I think it depends highly on the population distribution. As far as I check, this urban lengend comes from Kendall's Advahced Theory of Statistics. But the original sentence just says as below.

... For some distributions, particularly that of the mean, quite a satisfactory approximation is give by low values of n, say n > 30.... (On p.341 in the 6th edition.)

If I paraphrase the above sentence, I think that the author just says something like that "The CLT works for some statisitcs from some population distributions even for very small sample size (like n= 30)". The following is a naive example. If the population distribution is uniform, the CLT approximation seems well even for n = 5. But the population distribution is highly heavy tailed, the CLT appoximation does not work even for n = 100.

Random Reset(1111111);
y1 = J(5, 10000, Random Uniform());
y2 = J(100, 10000,Random SHASH(0.1, 0.1, 2, 3));
dt = As Table(VMean(y1)`||VMean(y2)`);
dt << Distribution(Y(1::2));

At third, the Wilcoxon's test is generally not a test for the difference between two means or two medians. So, if you are interested in the difference of means or medians, I think you should not use the Wilcoxon' test. The following is a very artificial naive example. The sample medians are equal, but the Wilcoxon's test is highly significant.

y1 = J(1000,1,Random Lognormal(0,1));
y2 = -y1;
y1 = y1 - Median(y1);
y2 = y2 - Median(y2);
dt = As Table( (J(NRow(y1),1,1)||y1)|/ (J(NRow(y2),1,2)||y2), <<Column Names({"G", "Y"}));
Column(dt,1) << Modeling Type("Nominal");
dt << Oneway( Y( :Y ), X( :G ), Quantiles( 1 ), Box Plots( 1 ), Wilcoxon( 1 ) );

At forth, from the point of veiw for statisitcal modelling, the flow chart itself is criticized in section1.1 in McElreath, R. (2020) Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd ed.), CRC Press.

In addition, if you judge Yes/No in the flowchart by some other statistical tests, these kind of pre-tests have also been often criticized. For an example, the checking homogeneity of variances by a statistical test before two-sample t-test or one-way ANOVA is said as "To make the preliminary test on variances is rather like putting to sea in a rowing boat to find out whether conditions are sufficiently calm for an ocean liner to leave port! (p.333, Box, G.E.P. (1953). Non-Normality and Tests on Variances). And, even if the pre-tests are statisitically valid, we cannot strongly accept H0 when p-values are large.

yusuke_ono

Thank you for your advice, @russ_wolfinger -san

About 20-30 years ago, when I was an instructor for basic statistical analysis using Base SAS and SAS/STAT, I recommended to SAS users that checking non-normality by UNIVARIATE, and checking homogeneity of variances by F-test in TTEST (and use the Wilcoxon for non-normal data, and use Welch's test in the case variances are different). This is partially because TTEST outputs the results of t-test for means and F-test for variances in the same page. My memory is rusty, but SAS/ASSIST also may have used the same kind of the flow-chart or cookbook logic.

But I now come to think that there is no statistical reason for this kind of rituals or customs.

However, in the 2016 p-value statement, it is said that “Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data ([…cut by Ono…]) would be equal to or more extreme than its observed value.” I think this explanation partially does not match to at least t-test. The statistical model for t-test was originally based on the normal distributions. But even if data is non-normal, (i.e. the data departs from the original assumed models), p-value can be large (when sample means are almost equal).

As you know, the randomized clinical trials usually do not use random sampling at all, but only randomization. So, I think we cannot give a valid statistical reason for using t-test in RCT from the random sampling framework (i.e. “population models” in Lehmann’s terminology).

russ_wolfinger

A couple classic references with competing views on randomization from two top-tier applied statisticians: Kempthorne: https://www.jstor.org/stable/2281178 Harville: https://www.jstor.org/stable/2683676