Modelling Guidance for Customer

wendytseng

I received a question from a (JMP Pro) customer who is running an electroplating experiment.  He has a designed experiment where he has varied several inputs with the goal of building a model to achieve a target outcome.  

 

He is struggling to build a good predictive model.  He says that when he uses the model to predict new runs, it does not do well.   

 

The model seems to change with respect to complexity (# of terms) very easily with the removal of a couple points. 

 

This sounds like a problem that could benefit from a one-on-one consultation with his data and his model but the very specific thing he asked was (I am paraphrasing a bit from our conversion):  "Recognizing that cross-validation methods are "doing this" already…is there an automated way to generate a model for each of the possible subsets of the experimental runs. I want to compare the # of terms (and which terms) for each of the models."

 

Questions that popped up in my mind were:

• Are you overfitting the model?
• Is your measurement system good? (He thinks it is)
• Are you able to control the inputs precisely [enough]?

What are additional questions should I be asking?  How can I be helpful to this customer?  

 

Thank you in advance!

wendytseng

I'm responding to my own post!  Further evidence I was never the popular kid...

 

I had a chat with @tom_donnelly and wanted to post my response to the client here.

 

"Thank you for spending time with me to explain your challenge.

 

I have done some outreach in the JMP technical community and the short answer to your question “Does JMP have an automated way to generate a model for each of the possible subsets of the experimental runs” is no.

 

However, I did receive some advice in some of my discussions that may be helpful to you.

 

I believe you are working with data from a designed experiment, correct?  Did you use a custom design?  How many runs did you have for error?

 

If this is the case, then the experiment that you designed…is a parsimonious set of runs to fit a hypothesized model (unless you had a lot of replicates).  The model, by nature, will be very sensitive to the data collected for each run because the design was created to answer the question (hypothesized model you specified) in as few runs as possible.  If your model is not doing a good job of fitting additional “test/validation” runs, then it may make sense for you to add to your original design to fit a higher order model.  Of course, the model may not be doing a good job predicting for other reasons, too (e.g. there are factors that are important that you are not controlling for, it is difficult to control your inputs, the measurements system has a lot of noise, etc.)

 

I recall you saying that you refit your model with additional runs.  Were these runs in addition to your original design?  Or were you fitting your model as you were collecting the data from the experiment? 

 

If the answer is the latter (is there a practical reason you are doing this?) – you could consider blocking your design.  For example, the first set of runs (first block) could be to estimate the main effects.  The second set of runs could be to estimate the interaction terms, etc.  That way – the runs are set up in each block so that they are estimating a full model.

 

Let me know if you would like to discuss further."

 

@tom_donnelly also mentioned that the customer may want to consider if transformations may be relevant in the model building but I did not mention it in my email (will wait until we chat live).