ANOVA in BIG data

IkjuShin

This is a question about a very common case.
The solution may or may not be very simple.
I'd like to hear your opinions and experiences.

1) This commonly occurs when ANOVA is applied to BIG data (N is very large).
2) (Other things being equal) Pvalue gets smaller as N increases.

Phenomenon
If there is a lot of data and the dispersion is small, it is not significant in practice, but is judged to be significant by ANOVA.
--> In practice, it is common to make errors by only looking at the ANOVA results.

How should I explain it?
Method 1: Practical differences rather than statistical significance (differences in confidence intervals, overlap, etc.)
(within-group variation is very large) is more important, so it is not of statistical significance.

Method 2: Look at Rsquare (Regression Variance / Total Variance), the value is low.

Method 3: In this case, ANOVA is not suitable.

Any other ideas?

 

 

Chandramouli_R

hi @IkjuShin 

I have tried it to explain it in a customer facing non-stat language as follows: 

In this analysis, we found a statistically significant difference between the two groups with a very small p-value. However, given the large sample size, it’s important to assess the practical significance of these findings. The R-squared value is quite low, indicating that the model explains only a small fraction of the variance. Additionally, the effect size is small, suggesting that the difference, while statistically significant, may not be practically meaningful. By examining the confidence intervals and visualizing the data distributions, we can better understand the extent of overlap and make a more informed judgment about the practical significance of the results.

______________________________broader understanding of this problem________________________________

This question addresses a common issue encountered when performing ANOVA on large datasets. Here are some points and methods you can consider when explaining this phenomenon:

Explanation of Phenomenon:

When the sample size (N) is very large, even small differences between groups can become statistically significant, leading to potentially misleading conclusions. This happens because the power of the test increases with larger sample sizes, making it easier to detect even trivial differences. Thus, the p-value becomes very small, suggesting significant differences where none may be practically meaningful.

Methods to Address and Explain the Issue:

Method 1: Practical Differences

Explanation: Emphasize the importance of practical significance over statistical significance. Statistical significance does not necessarily imply a meaningful or practically significant difference. Look at the confidence intervals and check for their overlap.

Practical Application: When the within-group variation is large, the practical difference may not be significant despite statistical significance.

Method 2: R-squared Value

Explanation: Use the R-squared value to assess the proportion of the variance explained by the model. A low R-squared value indicates that the model explains only a small portion of the variance, suggesting that the differences might not be practically significant.

Practical Application: Highlight cases where the R-squared value is low, indicating that the model does not explain much of the variability in the data.

Method 3: Suitability of ANOVA

Explanation: In cases where the sample size is very large and the differences are minor, ANOVA might not be the most suitable test.

Practical Application: Consider alternative methods or additional analyses to assess practical significance, such as effect size measures or confidence intervals.

Therefore the additional Ideas will be to:

Method 4: Effect Size

Explanation: Report effect sizes (e.g., Cohen's d) along with p-values. Effect size provides a measure of the magnitude of the difference, which can be more informative than p-values alone.

Practical Application: Use effect size to communicate the practical significance of the findings. A small effect size indicates that the difference, while statistically significant, may not be of practical importance.

Method 5: Graphical Representation

Explanation: Use graphical methods like violin plots or density plots to visually inspect the data distribution and differences between groups.

Practical Application: Visualizations can help convey the extent of overlap between groups and provide a more intuitive understanding of the practical significance.