772854_en-US

liliconcom

Stepwise regression issue

I encountered the following issue while selecting a model using JMP's Fit Model - Stepwise function.

1. If the interaction term is significant but the main effect is not, the main effect is removed from the model and the interaction term is also removed.
2. We understand that if an interaction or higher order term is significant, we should keep the main effect even if it is not significant.
3. Even if we use AIC/BIC criteria in addition to the P-value threshold, main effects are removed during the backward selection process.

I would be grateful if you could let me know any solutions or recommendations you may have regarding these issues.

This occurred in Y1 when considering four factors with only first-order interaction terms using the attached data.

JMP Step-by-Step Selection	6th step

The interaction term Factor 1 * Factor 4 is still significant, but Factor 1 is not significant and has been removed.

Re: Stepwise regression issue

The rules in Stepwise can be confusing. The issue you have run into is the "Combine" rules. From the JMP Help:

Combine

Calculates p-values for two separate tests when considering entry for a term that has precedents. The first p-value, p1, is calculated by grouping the term with its precedent terms and testing the group’s significance probability for entry as a joint F test. The second p-value, p2, is the result of testing the term’s significance probability for entry after the precedent terms have already entered into the model. The final significance probability for entry for the term that has precedents is max(p1, p2).

Tip: The Combine rule avoids including non-significant interaction terms, whose precedent terms can have particularly strong effects. In this scenario, the strong main effects might make the group’s significance probability for entry, p1, very small. However, the second test finds that the interaction by itself is not significant. As a result, p2 is large and is used as the final significance probability for entry.

Caution: The degrees of freedom value for a term that has precedents depends on which of the two significance probabilities for entry is larger. The test used for the final significance probability for entry determines the degrees of freedom, nDF, in the Current Estimates table. Therefore, if p1 is used, nDF equals the number of terms in the group for the joint test, and if p2 is used, nDF equals 1.

The Combine option is the default rule. See Models with Crossed, Interaction, or Polynomial Terms.

 

So now back to my comments. Because of the two tests that are done with the Combine rules, the non-significant term is driving the removal of the interaction. One way to avoid this is to change your Rules to be "Restrict". Here are the results when doing that:

I think these are the results that you expected.

Re: Stepwise regression issue

Starting with JMP 18, there's an option to respect heredity in the Stepwise platform

 

.

 

Re: Stepwise regression issue

Hi Dan, This is the result I was looking for. Thank you for suggesting the appropriate option. However, I am still confused about the options related to stepwise regression. "Restrict" is defined in the manual as "Restricts the terms that have precedents so that they cannot be entered until their precedents are entered." This seems to be a different rule than including main effect  if the interaction is significant. Additionally, the "whole effects respecting heredity" option that Don mentioned shows different results. Could you please provide a more detailed explanation to help me understand better? Thank you so much again. 

Re: Stepwise regression issue

@HyeWonJung using Combine, Restrict, No Rules and Whole Effect in models that have categorical terms with more than two levels is a good way to show what is happening. As an illustration, fit the following model using the Cars 1993 data set. Vehicle Category has five levels with the following precedence order:

1. {Van&Large&Midsize&Sporty&Compact–Small}
2. {Van–Large&Midsize&Sporty&Compact}
3. {Large&Midsize–Sporty&Compact}
4. {Large–Midsize}
5. {Sport–Compact}

This is how Stepwise builds one degree of freedom contrasts. It the same approach that Partition uses to create splits. If you run Partion with Highway Mileage as the Y and Vehicle Category as the only X, the splits will occur in exactly this order. Personally, I find that contrasts created in this fashion often lack interpretability and prefer the way Generalized Regression creates one degree of freedom contrasts based on the individual levels. I'll refer to these contrasts as C1 – C5.

Change Stopping Rule to P-value Threshold and set Prob to Enter to 0.40. Keep Direction set to Forward.

Start with No Rules. Click Step until no more terms are entered. Terms are entered in the following order:

1. Weight
2. Vehicle Category (C2) * Weight
3. Vehicle Category (C1) * Weight
4. Vehicle Category (C2)
5. Vehicle Category (C4) * Weight
6. Vehicle Category (C1)

Pretty straight forward. Find the smallest p-value regardless of precedence and keep adding until nothing outside of the model has a p-value smaller or equal to 0.4.

Click Remove All and repeat for Combine

1. Weight
2. Vehicle Category (C1) * Weight AND Vehicle Category (C1)
3. Vehicle Category (C2)
4. Vehicle Category (C4) AND Vehicle Category (C3)
5. Vehicle Category (C5)

Notice that in steps 2 and 4 two effects are added simultaneously. In each case p-values are calculated separately for both terms. In the case of step 2 the first term added is an interaction whereas in step 4 it is a contrast. Also note that Vehicle Category (C2) * Weight is missing from the model even though it was the second term added in the No Rules case. That's because when tested individual Vehicle Category (C2) was significant but Vehicle Category (C2) * Weight was not once Vehicle Category (C2) was in the model. This is Combine in action.

Now do the same for Restrict

1. Weight
2. Vehicle Category (C1)
3. Vehicle Category (C1) * Weight
4. Vehicle Category (C2)

This is similar to Combine except terms are added one at a time. The contrasts C3 and C4 were not added with Restrict because the p-value for C3 (precedent to C4) was too large to enter the model. However, Combine calculates the p-values with both C3 and C4 in the model and for C4 after C3 has been added to the model. Both of which are smaller than the 0.4 cutoff.

Finally, Whole Effects

1. Weight
2. Weight*Vehicle Category (all contrasts)

Here whole effect refers to all the contrasts in any effect and not statistical heredity. In this case, the interaction has a smaller p-value and once added to the model makes the Vehicle Category main effect p-value too large to consider.

If you did this once more with Whole Effects Respecting Heredity, the main effect would be added with the interaction at step 2.

Hope this makes sense. Let me know if you have questions.

 

 

Re: Stepwise regression issue

My comments will apply to the situation you have: all continuous variables. When you have categorical factors, it gets slightly more complicated, but I believe Don has pointed that out well in his longer post. By the way, I had not noticed the new option that Don pointed out. I think the JMP help describes it quite well:

As for Restrict, it restricts interactions (and quadratics -- terms that have precedents) so that they cannot be entered until the precedents are entered. In other words, an interaction cannot be entered into the model until the main effects have entered. In your situation, Factor 1 and Factor 2 both entered the model, thus Factor 1*Factor 2 could be assessed for entry. However, unlike Combine, Restrict is only conducting one test on that term rather than two tests.

Re: Stepwise regression issue

Thank you @dobudo @DonMcCormack !!

I've dive into the each options thanks to your explanation and understood the mechanism of each of option