Modelling Guidance for CustomerI received a question from a (JMP Pro) customer who is running an electroplating experiment. He has a designed experiment where he has varied several inputs with the goal of building a model to achieve a target outcome.
He is struggling to build a good predictive model. He says that when he uses the model to predict new runs, it does not do well.
The model seems to change with respect to complexity (# of terms) very easily with the removal of a couple points.
This sounds like a problem that could benefit from a one-on-one consultation with his data and his model but the very specific thing he asked was (I am paraphrasing a bit from our conversion): "Recognizing that cross-validation methods are "doing this" already…is there an automated way to generate a model for each of the possible subsets of the experimental runs. I want to compare the # of terms (and which terms) for each of the models."
Questions that popped up in my mind were:
- Are you overfitting the model?
- Is your measurement system good? (He thinks it is)
- Are you able to control the inputs precisely [enough]?
What are additional questions should I be asking? How can I be helpful to this customer?
Thank you in advance!
Re: Modelling Guidance for CustomerI'm responding to my own post! Further evidence I was never the popular kid...
I had a chat with @tom_donnelly and wanted to post my response to the client here.
"Thank you for spending time with me to explain your challenge.
I have done some outreach in the JMP technical community and the short answer to your question “Does JMP have an automated way to generate a model for each of the possible subsets of the experimental runs” is no.
However, I did receive some advice in some of my discussions that may be helpful to you.
I believe you are working with data from a designed experiment, correct? Did you use a custom design? How many runs did you have for error?
If this is the case, then the experiment that you designed…is a parsimonious set of runs to fit a hypothesized model (unless you had a lot of replicates). The model, by nature, will be very sensitive to the data collected for each run because the design was created to answer the question (hypothesized model you specified) in as few runs as possible. If your model is not doing a good job of fitting additional “test/validation” runs, then it may make sense for you to add to your original design to fit a higher order model. Of course, the model may not be doing a good job predicting for other reasons, too (e.g. there are factors that are important that you are not controlling for, it is difficult to control your inputs, the measurements system has a lot of noise, etc.)
I recall you saying that you refit your model with additional runs. Were these runs in addition to your original design? Or were you fitting your model as you were collecting the data from the experiment?
If the answer is the latter (is there a practical reason you are doing this?) – you could consider blocking your design. For example, the first set of runs (first block) could be to estimate the main effects. The second set of runs could be to estimate the interaction terms, etc. That way – the runs are set up in each block so that they are estimating a full model.
Let me know if you would like to discuss further."
@tom_donnelly also mentioned that the customer may want to consider if transformations may be relevant in the model building but I did not mention it in my email (will wait until we chat live).
Re: Modelling Guidance for CustomerI like Tom's answer to this. I think it is the best. However, if there is no time or budget for more runs then I have also had some luck using Generalized Regression to get more predictive models on DOE data in my past life. Peter Hersh shared an awesome JMP Pro demo with me that is so close to what I did it is scary:) Pete, can I bump this to you?
Re: Modelling Guidance for CustomerHi Wendy,
I have attached my journal for the example that Jason is talking about. It is very similar situation where they have run a DSD and are trying to build a predictive model. If you use Stepwise then fit least squares (standard JMP) you get a very overfit model that ends up being bad a predicting. With GenReg it works much better. I am out of town this next week, so I can help explain it better, but all of the scripts are saved in the data tables. It makes a nice case for GenReg vs. Stepwise/fit least squares. Hope this is helpful.
Re: Modelling Guidance for CustomerIt might be worth checking if this experiment includes mixture factors. If so, we shouldn't be attempting to simplify the model in the way we would for an experiment involving just process factors. Instead try going with predictions using the design model.
Best,
Malcolm
Re: Modelling Guidance for CustomerThank you!
