Cringeworthy statistics statement #2 explained: Statistical significance vs practical importance
We've all heard someone say something that isn't "right." Sometimes you just let it go, either because it isn't important enough to challenge, or you don't want to cause conflict with the speaker.
This is the second "cringeworthy statistics" blog series installment. In general, here's how it works: First, I post some kind of statistics-related question in a JMP Community Discussions thread. Community users are encouraged to reply. After about a week, I give my take on what is wrong with the statement in a blog post like this.
Here is a table of discussion topics and blog posts to date. Many thanks to those who reply to each discussion post (and/or add comments to the blog!)
So let's take a look at the second cringeworthy statement I posted...
My take on Cringeworthy #2
Recall in our scenario that a study had been done to try to determine why yield had fallen in a production process. A t-test was performed to study differences between two suppliers' subassemblies. Indeed, the t-test showed a very low p-value, giving high confidence of a difference in the means of the parts from the two suppliers.
The team manager wanted to jump at the fact that the p-value was small. He wanted to immediately start a study on why the two population means were different. (Perhaps he was driven by past experience that a difference between these vendors had once caused a problem.) But what the manager didn't mention was whether the difference was "practically important."
In my experience, people often confuse "statistical significance" with "practical importance."
- Statistical significance is simply a mathematical statement that there we can detect some amount of difference in the means of the two populations at some level of confidence. This is nice, but it says nothing about whether that difference is important to us.
- Practical importance is based on your judgement, experience and wisdom. Maybe the statistical test detects a difference of 0.1 units between the two treatments. Is a difference of 0.1 important to your application? If not, then perhaps devoting resources to discovering the source of this difference would be better spent looking for other sources that might lead to poor yield.
Maybe it is unfortunate that the term "significance" can be interpreted as "importance" in the English language. I don't know of a better word to use than significance, but it can be misleading to some people.
Moral of this story: Make sure to understand the difference between statistical significance and practical importance when interpreting statistical analytic results!
Now, please come on over to Cringeworthy Discussion #3!
"It's statistically significant! We should go devote resources to the problem!"
Does this statement make you cringe? Have a look at why I think this is a cringeworthy statement, in this summary of my second discussion topic.
Re: Cringeworthy statistics statement #2 explained: Statistical significance vs practical importanceI am using discernible difference and practical importance now. It feels better. Hopefully, others will follow. You are right. This is an English language problem. Synonyms for significant are important, crucial, critical, meaningful, critical, serious, and vital. So the misuse of the term makes sense.
How do we get a significant percentage of the population to use a more appropriate expression, utterance, term, or word?
I am not so good with words. I'll stick numbers. It took me way too long to write that last sentence. I couldn't choose the right expression, utterance, term, or word. So, I used them all.
Thanks again, @JerryFish,
Rob
Re: Cringeworthy statistics statement #2 explained: Statistical significance vs practical importanceLOL Thanks @DirtyDataCoLLC !
I also heard from @Craige_Hales, who sent this link regarding the same discussion topic. In it, the author recommends "Discernible Difference" instead of "Statistical Significance," because of the misleading implications of SS. Makes a lot of sense to me!
Re: Cringeworthy statistics statement #2 explained: Statistical significance vs practical importance
I know many people who are going to enjoy your explanation of statistically significant vs. practical importance. A few years ago, the FDA did not share your wisdom, and an entire section of a BLA had to be rewritten. The FDA opinion took a year to revise.
Although, someone could argue that one year is trivial if we think of time on the human evolutionary scale. So I guess it was not practically significant.
