Am I going crazy or How is that possible? Help troubleshoot post-hoc tests

Question

I've ran a full-factorial ANOVA model on some data I've collected and the output looks something like this:

This look ok, but when I go to investigate some of the interaction effects, It looks like there is something going on there according to LSMEANS Plots, for instance:

Then when I select LSMEAN Tukey's HSD test it gives me signficant differences between TRTxTIME! what is going here?

If this interaction isn't signficant in the effects tests, why would it then should signficance on TRTxTIME response? Am I going crazy...? Please help me to understand why this is happening.

Peter_Bartell · Answer

I'll just ask one more question to pile on to my colleagues' @Mark_Bailey and @KarenC's counsel, which I agree with. When you wrote in your original question your wrote "...ran a full factorial ANOVA..." does this mean you EXECUTED a full factorial EXPERIMENT? Then analyzed the data accordingly in the JMP Fit Model platform? And not just gathered up a bunch of combinations of your factors at willy nilly levels and combinations, THEN picked the 'Full Factorial' option from the Fit Model Macros drop down window? Reason being if you did NOT run a full factorial EXPERIMENT then in all liklihood you've got loads of multicollinearity wrt to the effects you are trying to estimate buried in the treatment combinations you ran that make estimating effects, model terms, etc. much more problematic within the context of Standard Least Squares.

KarenC · Answer

Agree with Mark's advice - also add in your subject knowledge to the data interpretation.  Confirmation/follow-up is always good.

Mark_Bailey · Answer

You might be going crazy. I can't help with that problem.

As to your analysis, you are confusing different tests as the same test. The effect test is comparing the model with and without that term. That test is based on 8 degrees of freedom. It says that the interaction term overall is not significant and can be eliminated without significant loss to the model. The leverage plot indicates essentially the same result because it is also based on the whole term effect. The LS Means plot is consistent with this conclusion because the change in the response from Time=24 to Time=48 is practically the same for all TRT levels except the one at the top of the plot. Notice the width of the confidence intervals.

Now the second test (Tukey comparison) is simultaneously comparing many pairings of factor levels. On the one hand, there is a great deal of overlap in the letter groups. On the other hand, some of these comparisons appear to be significant. If they mean something to you, then you could investigate those pairings separately. Perhaps a confirmatory test run.

So you have a choice: delete the term based on fhe negative effect test and ignore the interaction effect in subsequent statistical tests and modeling or accept the positive comparison tests and perform further runs to confirm the apparent differences. Confirmation runs are a good idea regardless of your choice.

Another idea is to follow a standard course of model reduction and then interpret the final result. This model is over-fitting the data and that leads to artificially low RMSE and standard error estimates.