Hypothesis Test Contradictory Results Interpretation

Question

Dear JMP Community,

I am trying to figure out the contradiction between 2 hypothesis system per below done at significance level of 0.05.

Looking at the first hypothesis system

2 sided Test

Ho: Mu=0 , H1: Mu not equal 0,  P(value) = 0.0705

Statistical Conclusion : Failed to Reject Ho hence Mu=0

However, if we look at the third system

Lower side Test

Ho: Mu >= 0 , H1:Mu < 0.  P(value) = 0.0352

Statistical Conclusion : Accept H1 hence Mu < 0.

In this case, there is a contradictory Statistical Conclusion between these 2 Hypothesis system.

The 2 sided test says Mu=0

The Lower side Test says Mu <0

How do we interpret this ?

Thanks to advise on this matter.

B.r,

Chris

MRB3855 · Answer

Hi @Zappy : In the spirit of @dlehman1  comments, confidence intervals may be a better way of understanding this. Sure, p-values and confidence intervals are derived from the same principles and there is a one-to-one correspondence between them...but let's set that aside.

Let's assume for our purposes that alpha=0.05.

1. You will reject [Ho: Mu=0]  in favor of [H1: Mu not equal 0], if the 95% confidence interval for Mu does not contain 0. This interval has alpha/2 =0.025 in each tail.

2. You will reject [Ho: Mu >= 0]  in favor of [H1:Mu < 0] if the 95% upper bound on Mu is less than 0. This upper bound has alpha in the upper tail.

The upper bound of the confidence interval in 1 is greater than the upper bound in 2 (since 1 has less probability in the upper tail than 2). Therefore, the two-sided test may reject while the 1-sided doesn't.

dlehman1 · Answer

You are showing a fundamental misconception - one that has led many (myself included at times) to call for an end to null hypothesis significance testing (NHST).  You should never interpret the test in a binary fashion.  What you have reported would be correct with binary thinking:  in a two-sided test, the null hypothesis was not rejected and in a one-sided test it is rejected (where the p value is exactly half of the p value for the two sided test).  With binary thinking, you do not reject the null for the two sided test but you do for the one sided test.  But the evidence is exactly the same in both tests.  What has changed is your willingness to commit type 1 errors.

What you should conclude is that the chance of obtaining a mean at least as far away from zero as 1.2667 standard errors - in either direction - is around 7% and the probability of obtaining a mean at least as far away from zero as -1.2667 in a negative direction is around 3.5%.  If you insist on using .05 as a binary cutoff, then yes you will reach different conclusions.  While this may seem puzzling to you, it only reflects a poor way to use the p value.  When you fail to reject the null, you should NEVER say it is true.  You have insufficient evidence to reject it.  Since the one-sided test is less demanding (since you only care about seeing if the evidence is sufficiently low to feel safe saying it is less than zero in this case), you have sufficient evidence using that binary cutoff value of 0.05.

I think the use of such binary cutoff p values is something that should be abandoned.  I do find p values useful, but not in that way.  They do indicate the strength of evidence that the data provides relative to a particular model, but the p value - by itself - should never tell you whether that evidence is sufficiently strong to reject that model.  In your case, the two-sided test and one-sided test are different models, hence the sufficiency of the evidence changes if you apply the same binary cutoff value.  But I would add:  where did that binary cutoff value come from? (unfortunately, the answer is often that it came from an introductory statistics text, and one that has outlived its usefulness).

Byron_JMP · Answer

So, just for clarification 1-0.9648=0.0352, and 0.0352*2=0.0704 (the values in JMP aren't rounded like they are here)

One sided tests

What's the probability of being less than t, pretty small. (the dark blue area)

What's the probability of being greater than t, pretty large. (all the area to the right of the dark blue area)

Two sided test

What's the probability that t is in side the 95% confidence intervals, pretty small (0.07) (in the white space on the figure)

In common language, not perfect stat-speak, it's like this:

if I say that -1.2667 is different from 0 what's the probability I'm wrong.   prob>|t|

If I say that -1.2667 is greater than 0 what's the probability I'm wrong.     prob>t

If I say that -1.2667 is less than 0 what's the probability I'm wrong.          prob<t

Perhaps someone more versed in specific stat-speak can add null to this?  (pun)