ANOVA in BIG dataThis is a question about a very common case.
The solution may or may not be very simple.
I'd like to hear your opinions and experiences.
1) This commonly occurs when ANOVA is applied to BIG data (N is very large).
2) (Other things being equal) Pvalue gets smaller as N increases.
Phenomenon
If there is a lot of data and the dispersion is small, it is not significant in practice, but is judged to be significant by ANOVA.
--> In practice, it is common to make errors by only looking at the ANOVA results.
How should I explain it?
Method 1: Practical differences rather than statistical significance (differences in confidence intervals, overlap, etc.)
(within-group variation is very large) is more important, so it is not of statistical significance.
Method 2: Look at Rsquare (Regression Variance / Total Variance), the value is low.
Method 3: In this case, ANOVA is not suitable.
Any other ideas?
Re: ANOVA in BIG datap-value of ANOVA tells you if the means are statistically different. Like you mention if you have a lot of observations, it becomes easier to achieve a low p-value for a given mean difference and stdev.
Looking at the Rsq will tell you how much of the variation is explained by that mean difference. So it's important to look at both metrics. In your example, the mean difference accounts for 16.8% of the variation, and the means of those two distributions are very likely to be statistically different even though the difference is small.
Re: ANOVA in BIG dataIn cases like this you want to add equivalence testing after the ANOVA. This can happen a lot in high volume situations. As Mark says, with a lot of data, you can detect very small differences. The question here becomes, can you do anything about it? Or, at what level do differences become so small they’re no longer worth chasing.
MRe: ANOVA in BIG datahi @IkjuShin
I have tried it to explain it in a customer facing non-stat language as follows:
In this analysis, we found a statistically significant difference between the two groups with a very small p-value. However, given the large sample size, it’s important to assess the practical significance of these findings. The R-squared value is quite low, indicating that the model explains only a small fraction of the variance. Additionally, the effect size is small, suggesting that the difference, while statistically significant, may not be practically meaningful. By examining the confidence intervals and visualizing the data distributions, we can better understand the extent of overlap and make a more informed judgment about the practical significance of the results.
______________________________broader understanding of this problem________________________________
This question addresses a common issue encountered when performing ANOVA on large datasets. Here are some points and methods you can consider when explaining this phenomenon:
Explanation of Phenomenon:
When the sample size (N) is very large, even small differences between groups can become statistically significant, leading to potentially misleading conclusions. This happens because the power of the test increases with larger sample sizes, making it easier to detect even trivial differences. Thus, the p-value becomes very small, suggesting significant differences where none may be practically meaningful.
Methods to Address and Explain the Issue:
Method 1: Practical Differences
Explanation: Emphasize the importance of practical significance over statistical significance. Statistical significance does not necessarily imply a meaningful or practically significant difference. Look at the confidence intervals and check for their overlap.
Practical Application: When the within-group variation is large, the practical difference may not be significant despite statistical significance.
Method 2: R-squared Value
Explanation: Use the R-squared value to assess the proportion of the variance explained by the model. A low R-squared value indicates that the model explains only a small portion of the variance, suggesting that the differences might not be practically significant.
Practical Application: Highlight cases where the R-squared value is low, indicating that the model does not explain much of the variability in the data.
Method 3: Suitability of ANOVA
Explanation: In cases where the sample size is very large and the differences are minor, ANOVA might not be the most suitable test.
Practical Application: Consider alternative methods or additional analyses to assess practical significance, such as effect size measures or confidence intervals.
Therefore the additional Ideas will be to:
Method 4: Effect Size
Explanation: Report effect sizes (e.g., Cohen's d) along with p-values. Effect size provides a measure of the magnitude of the difference, which can be more informative than p-values alone.
Practical Application: Use effect size to communicate the practical significance of the findings. A small effect size indicates that the difference, while statistically significant, may not be of practical importance.
Method 5: Graphical Representation
Explanation: Use graphical methods like violin plots or density plots to visually inspect the data distribution and differences between groups.
Practical Application: Visualizations can help convey the extent of overlap between groups and provide a more intuitive understanding of the practical significance.
Re: ANOVA in BIG dataI think @mzwald is pointing to the important direction. To me, "differences" is not well defined in the posed questions.
Follow along Mark's comment, the next question will probably be: is the "difference" that ANOVA is designed to address the same as what actually concerns one?
Re: ANOVA in BIG dataAdding to already insightful comments, here are some deeper questions to consider:
What are the real objectives of the study?
To explain or predict? Stat methods / hypothesis testing / ANOVA / t-test / confidence limits / equivalence testing are good for explaining, whereas predictive modeling / machine learning with cross validation is good for predicting. The two are not mutually exclusive.
Are there any additional data / covariates available? With modern measurement technology, large data like these with only a single treatment measured are becoming increasingly uncommon. Extra variables / features can make the stat and predictive modeling much more effective and explain much more variability.
Are the measurements truly independent? e.g. If there are blocking variables / batch effects, then mixed models become very relevant. If this is actually a split-plot design and treatments are assigned at the whole plot level, power may not be nearly as high as originally thought.
Re: ANOVA in BIG dataGreat and Practical advice !!!
-Sample size
-RSquare
-Effect size(Sum of square)
-Within variation
-Confidence/Prediction interval
-Visualization
It seems that all these things need to be considered from a practical point of view.
Some people seem to 'unconsciously' mistake ANOVA as a test for differences in raw data rather than a test for differences in means.
