nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Why Is This a Cringeworthy Statistics Statement? #1

JerryFish

I'm going to start a few discussion threads over the next few weeks, and I invite any JMP user to chime

in with their thoughts. These discussions will revolve around "Cringeworthy Statistics Statements."

Please consider contributing. After a few days, I'll summarize thoughts in a blog post, and introduce another Cringeworthy Statement!

Here is Cringeworthy Statement #1:

We run a t-test to compare the means of two populations. We want 95% confidence in the results. We run the test, and the p-value comes back at 0.63. We make the statement "Since p is not less than 0.05, we conclude that there is no difference in the means of these populations."

Why is this statement Cringeworthy????

Accepted answers

All comments

dale_lehman

I agree with this example of cringeworthiness - classic misunderstanding of the absence of evidence not meaning evidence of absence. Please consider the classic, but less straightforward: "the 95% confidence interval ranges from x to y, meaning that I am 95% confident that the true effect is between x and y." I happen to believe it is fine to say that, even though I know the correct statement is that "95% of all random samples constructed this way will contain the true effect." Of course, we don't know if our sample is one of those 95% or one of the 5% that don't contain the true population effect, and it either does or does not. But, given the one sample we have, I think the best we can say is the first quoted statement, despite it technically not being correct.

I'd like to hear what other people think of this example. Statisticians love to point out how this wrong interpretation of the confidence interval is common, while I can completely understand it. And I confess to giving students the liberty to wrongly interpret it (though at least I point out what the correct interpretation is). The problem with the correct interpretation is that it doesn't leave us able to say anything about the sample we actually have.

P_Bartell

Where do I start? Sample size (none stated...might just be too small to give the signal a chance to rise above the noise)? What is the chosen alpha risk ('...95% confident in the results...' is a meaningless statement wrt to alpha risk. What does that statement even mean)? What is the null hypothesis (none stated)? What is the measurement error (known or even assumed)? What was the sampling plan (random, convenience, first/last off the line, stratified or some other plan)? What's the representation risk...not just the statistical risk? And on and on.

statman

I'll play...My favorite is actually quite simple...Using the phrase "statistically significant" applied anywhere without context. It's interesting that if you ask someone if they are more likely to believe something when the phrase statistically significant is added they answer yes without knowing anything about the comparisons being made.

JerryFish

Hey @P_Bartell ! Good to hear from you!

Thanks for the reply!! We'll put these ideas in my (growing) list of cringeworthy statements, and bring them up in future posts!

Stay well, my friend!

JerryFish

Thanks @statman ! More good fodder for future posts.

Keep the comments coming!!

+ No mention of a check for normality

Georg

To widen the discussion a little - to me this does not sound cringeworthy from the beginning. In my opinion it can still be a valid decision. Of course, as others pointed out already, there may be missing a lot of additional information to trust that decision. So to my eyes it depends on the context of the statement, and who put it. And what, by the way, would be the right statement with the same number of words?

Also, although I like sometimes discussions about right wording - terms are important, I don't like building up a wall between statisticians and process engineers that do not have that deep knowledge in statistics. But the latter know their data and processes. And as we can imagine, there are a lot of things that are even more important beneath the bare table of numbers. This could be environment of data sampling as well as consequences of the decision made, like @P_Bartell stated already. Decisions still have to be made by humans, not by machine.

P_Bartell

I am really fond of @Georg 's last sentence in his reply above. As a statistician working on process/product commercialization or corrective action teams for 20+ years in industry, I would often tell the engineers and scientists and managers I worked with, "The day you let me, a statistician, tell you how to run your processes is the day your road to ruin begins. You've forgotten more about your processes/products than I'll ever know."

Or as Walter Shewhart wrote (paraphrased by me) : "The long range contribution of statistics depends not so much on creating a bunch of statisticians as it does in creating a group of statistically minded engineers, scientists, production personnel and others."

brady_brady

In addition to other problems mentioned in previous posts,

1) When testing equivalence of means, the sidedness of the test must be specified a priori. If not otherwise stated, testing for mean equivalence usually implies a 2-sided test, in which case alpha, the 5% type I error probability, is the sum of the area in the upper and lower tails of the null distribution. In almost all cases, this probability is divided equally, so the tail area to which the p-value should be compared should be alpha/2, which is 0.025, not 0.05.

** EDIT: JMP takes care of this for you by doubling the tail it found using the critical statistic, but if you were doing this "by hand", you would end up rejecting for a single-tail area of < 0.025.

2) "95% confidence in the results" is cringeworthy because it misunderstands the main idea behind a hypothesis test. You can be 100% confident that, in running a hypothesis test at the 0.05 level of significance, you will have employed a certain procedure (if you've done things correctly). When the means of the 2 distributions are actually the same, over the long term (and assuming all assumptions hold), 5% of the time that procedure will produce a test statistic which results in a conclusion of differing means, due to sampling variability.

3) There are only 2 possible conclusions for a hypothesis test of this nature: 1) "The sample data provides (compelling) evidence that the null hypothesis is false" and 2) "The sample data does not provide compelling evidence that the null hypothesis is false". Notably absent is the statement "Based on the sample data we conclude that the null hypothesis is true." Generally when you want to prove something, you frame what you are trying to prove as the alternative hypothesis, hoping that the data supports rejection of the null in favor of the alternative. If the researcher is trying to demonstrate equivalence in means, then two one-sided tests would be more appropriate.

4) Others have mentioned this but it bears repeating: statistical significance and practical significance need not be, and usually are not, the same thing. Practical significance should be discussed and agreed upon from square one, if not square zero.

@dale_lehman mentions that "The problem with the correct interpretation is that it doesn't leave us able to say anything about the sample we actually have." I'll disagree, but I know where he is going with this. Every statistical technique has a particular question it is designed to answer. Particularly in the case of a null hypothesis that the data has failed to reject, one of the main issues non-statisticians (and Bayesian statisticians, for that matter) have with frequentist hypothesis tests is that a p-value answers a question that is an unnatural question to ask, namely: "Under the null hypothesis, what is the probability that I would observe results as extreme as, or even more extreme than, the results I actually did observe?". Bayesians will say that their methods answer more natural questions, but Bayesian techniques have their own issues--there is no panacea. There is no substitute for understanding which techniques are better equipped to answer certain questions, and choosing a technique that is well-suited to the question you are trying to answer. The recent(ish) backlash against the p-value, especially in the medical research arena, strikes me a bit like a backlash against forks when they're used as eye patches. A p-value is a tool, like any other. Use it the right way, for the right job, and you'll be fine... use it the wrong way and "you'll shoot your eye out!".

dale_lehman

I don't think my example really has anything to do with Bayesian vs Frequentist analysis or p values (at least directly). You are correct, of course, that saying I'm 95% confident the true value likes in this 95% confidence interval is incorrect: the confidence is in the procedure, not any one particular confidence interval. The true value either is or is not in that one particular interval.

But my point is that 95% is the best estimate I can give of whether or not my one particular interval is one of the 95% of "good ones" that contains the true value or one of the 5% that would be giving the wrong conclusion. You may find the wrong interpretation cringeworthy but I do not. In fact, I find the correct interpretation somewhat cringeworthy, as it is a bunch of words that makes most people's eyes water over. And, I think it contributes to the far worse (and more cringeworthy) practice of ignoring the uncertainty altogether and treating the point estimate as the true value.

PatrickGiuliano

I wouldn't call the interpretation: "I am 95% confident that the sample mean difference is not less than Lower CI and not more than Upper CI" 'wrong' per se, in fact I think it's a perfectly appropriate way of phrasing it that stands "correct" and easily phrased, but I would say that the interpretation is just "commonly misunderstood" @dale_lehman as you point out.

As @jules says in Decision Making with Data (STIPS) - Section 4.1 Estimation {and you can get this from the Transcript where I have screen grabbed the appropriate module reference below entitled "Demo: Understanding the Confidence Level and Alpha Risk"}

"We'll click Reset Samples, change the number of samples to 100, and click Draw Additional Samples. As we continue to draw samples, notice that the mean of the samples is close to the true mean. Also notice that close to 95% of the confidence intervals contain the true mean IQ value, and about 5% of the confidence intervals don't include the true mean."

"This is what we mean by the term "confidence." On average, for a 95% confidence interval, 95% of the time this interval method is used, the interval captures the true parameter value."

Of course, the inference is on the population, but it's based on the sample data. This is what we are doing when we are doing inferential statistics in general. The whole enterprise is about making inferences on the population based on the sample data that we have. Again, in the same Decision Making with Data Module 4, Watch Section 4.1 Introduction to Statistical Inference. Here, @mia_stephens speaks about this in a clear, precise and direct way.

PatrickGiuliano

To me it just doesn't reflect an adequate level of understanding, but at least it's more or the less the "correct interpretation."

A few folks discussed something along the lines of what I like to call the "p-value method" and the "confidence interval method" of running a T-test. I like to think of these two ways of analysis as in-essence the same thing and I will propose here (and please feel free to disagree with me) that they are in fact, different ways (of mathematically expressing) the same thing. JMP is very smart about giving you the statistical presentation both ways in most cases, and I really like that! Interpretation using confidence intervals (in the context of plain vanilla T-testing, non-inferiority testing, equivalence, and superiority testing) has always been more interpretable to me.

A dear friend of mine gave me a nice textbook reference that I think speaks to this:

Bluman, Elementary Statistics (7th edition), 2015