Fit Definitive Screening vs. Stepwise (min. AICC) for model selection

Question

I am trying to fit a model to a 17-run DSD with 4 factors. I am evaluating two methods of doing so: the 'Fit Definitive Screening' platform that is recommended by JMP documentation, as well as the older forward selection method with minimum AICC as the criterion.

Based on domain knowledge, I would have expected a quadratic effect for factor B which looks like a plateau, with the low level being significantly worse than the mid/high levels, which should be relatively similar. When I look at the results of the 'Fit DSD' algorithm, the main effects residual plot for factor B does in fact look like this. However, the fit DSD algorithm does not output any quadratic effects. I have tried altering the Stage 2 ratio from 1 to 0.1 and found no difference.

When I take the same dataset and use forward model selection with minimum AICC, I get the following model, which does include B^2 as a term.

Finally, when actually running the models made by Fit DSD vs. stepwise, I find that the overall p-value and adjusted R^2 of the stepwise-derived model to indicate superior fit (p = 0.0144 and r^2 = 0.438 for fit DSD, p = 0.0030 and r^2 = 0.615 for stepwise)

I repeated this for some other output parameters and found that in general, fit DSD is much more conservative than stepwise selection with regards to second-order terms. Is this true? If so, is it still recommended to use the model from fit DSD, even though it results in poorer model fit?

I have attached anonymized data.

Thank you.

Sample data.jmp

Mark_Bailey · Answer

These approaches and others are the bases for tools that help you but do not dictate one way or answer. The Fit Definitive Screening approach failed to find a suitable model in your case. JMP provides a plethora of tools based on myriad approaches so you do not have to limit your search to select the best model to just one approach.

JMP has all these tools because no approach has been shown to be superior in all cases.

Also, by analyzing your data with different approaches, you can build your confidence when they agree. When they disagree, you can investigate the reason for the disagreement and understand why the selected model is best.

Above all, you should validate your model. Use it to predict the response at settings not included in the original experiment. You should consider settings that predict the desired outcome and settings that predict an undesirable outcome. Your model should predict all responses accurately within the space of the experiment.

Victor_G · Answer

Hi @QW,

The "Fit Definitive Screening Design" is an analysis designed for DSD, as it emphasizes and supports the DOE principles of effect hierarchy, effect heredity, and effect sparsity : Principles and Guidelines for Experimental Design (jmp.com)

* In a first round, the Fit DSD platform tries to find the significant main effects, and create a model based on the identified main effects.
* In the second round, the residuals from the first model are analyzed to find and detect 2nd order terms (dependent from the main effects identified before, following the principle of effect heredity) like interaction terms or power terms.

By default the platform uses "strong heredity" principle (example: interaction A*B may only be analyzed and identified in the second round if both main effects A AND B are identified and significant in the first round), but you can have more "flexibility" in the modeling by de-checking the "Quadratic terms obey strong heredity" and "Interactions obey strong heredity". That will enable weak heredity, meaning that interaction A*B may be analyzed in the second round if main effect A OR main effect B is identified and significant in the first round for example. In your case, the weak Heredity does provide the same model. More infos here : Effective Model Selection for DSDs (jmp.com)

Please also note that DSD, even if powerful and interesting to detect 2nd order effects, are still screening designs, so you won't be able to detect and estimates all the possible effects, and the power related to quadratic terms or interactions terms is quite low compared to main effects for example, so it seems "normal" to have more difficulty in detecting quadratic effects :

If you think there might still be hidden terms that were not detected among the 2nd order effects, I would highly recommend you to augment your design with more runs (with the platform "Augment Design"), to have more confidence in your model (and possibly detect new quadratic effects and/or interaction effects).

The "Stepwise" selection approach is different, more oriented like a "model-agnostic" analysis, where the selection of the terms do not necessarily follow some of the DoE principles (even if you can "lock" some terms like main effects to respect effect heredity/hierarchy), but is focussed on optimizing a criterion (AICc, BIC, ...). This can result in more complex models, as the "safeguards" from other platforms like "Fit DSD" may not be present. If you want to really push the boundaries and explore other designs with the Stepwise platform, you can also click on the red triangle, and try "All Possible Models". You can specify the number of terms in the model, and JMP will try all possible combinations of terms to provide all possible models with the number of terms specified. It can give you some "inspiration" in the modeling, but the results should be taken with caution, as some terms may be entered in the model without any statistical significance (so a little bit of fine-tuning might be needed after) :

With the first model (some terms are added by default but their p-values are above the 0,05 threshold):

It's always interesting to try and compare different modeling options, and even more when domain expertise can guide you in the process. It can help you have ideas about important terms, and may help you in defining how to augment your DoE with new runs, based on a supposed model.

I don't think there is a right or wrong answer here, the key is really to compare and evaluate different models to get a good understanding of the case. Some methods are more conservative than others, but combining different modeling with domain expertise can help you have a broader view about what matters the most. And from then, plan your next experiments to refine your model, and prepare some validation points to be able to assess your model.

I hope this (long) answer will help you,

PS : You can also compare other models done with different platforms like Generalized Regression with different estimation methods (Best Subset, Pruned/2-stage/ Forward selection, ...).