Hi all, I designed a DSD with the following info: - Design: 7 continuous and 1 categorical factors, 22 run - Design Evaluation on power and correlation: very good - Fit model via Fit Definitive Screening: excellent * RMSE = 0.2803, RSq =0.997, PValue F = 0.0561) * Residual and Studentized Residuals are all good - Model structure: * Contains 5 main, 4 quadratic and 1 interaction factors - Model confirmation: try a few runs (points) around to Max desirability * some agree with prediction very well * some show large differences Due to resource constrain, we can run 5 to 10 confirmation runs. Please advise: - Potential causes of the difference say underfit or overfit? - Method or procedure to improved/optimized the model Thanks.

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Definitive Screen Design (DSD) Model confirmation

ZenCar

Hi all,

I designed a DSD with the following info:

- Design: 7 continuous and 1 categorical factors, 22 run

- Design Evaluation on power and correlation: very good

- Fit model via Fit Definitive Screening: excellent

RMSE = 0.2803, RSq =0.997, PValue<0.001)
Lack of Fit: no (F Ration 191, Prob > F = 0.0561)
Residual and Studentized Residuals are all good

- Model structure:

Contains 5 main, 4 quadratic and 1 interaction factors

- Model confirmation: try a few runs (points) around to Max desirability

some agree with prediction very well
some show large differences

Due to resource constrain, we can run 5 to 10 confirmation runs.

Please advise:

- Potential causes of the difference say underfit or overfit?

- Method or procedure to improved/optimized the model

Thanks.

Accepted answers

All comments

statman

This will be very difficult for us to help with. Realize the statistics you report have nothing to do with whether the model works into the future. Extrapolation is an engineering or scientific one and it depends greatly on how representative your experiment was of the future conditions. The definitive screening design is a strategy to handle design factors, not the noise (factors you are unwilling to control or manage). You say nothing about how you handled the noise. If the noise factors are held constant, then your inference space is too small and your model will likely only apply to the data in hand. For future experimentation, my advice is to spend as much time considering what you will do with the factors not included in the design structure (noise) as the ones you are.

ZenCar

Hi Statman,

The DOE was based on simulation software results. We found one factor level (-1,0,+1) did capture the full range of the representing variable, but physically (0, 0.5, 1) or (-1, -0.5, 0) should be the effective range for target response. So we can view this factor as a noise factor and use a same factor level in confirmation run.

Based on the additional info above, any advice please?

Thanks.

statman

Not sure I can help you. Are you saying the results from running the experiments in the simulation software are not repeatable in the simulation software? Or are you saying the results of the experiment in the simulation software are not repeating in "real life"?

I think there may be confusions to what noise is. It goes by a number of different names: Nuisance, background, et. al. Examples include: ambient environmental conditions, wear or degradation of materials and equipment, operator technique, in some case measurement error, lot-to-lot variation of materials, etc. These variables are not typically manipulated in an experiment and are either held constant (bad idea) or vary during the execution of the experiment. These need to be representative of future "conditions" for your experimental results to be applicable in the future (or for your model to predict the response variables in the future).

If you are using simulation software, the algorithm is already contained in the software. Do you not know it? I have no idea how your simulation software "simulates" noise.

ZenCar

Hi Statman,

Thanks for your patience.

I am familiar with the noise concept. I started by using Taguchi Method for many years. So in my case, we don't need to worry about noises.

In short, the DOE was based on the results from running simulation software, all looks good except model works poorly with some confirmation data.

To figure out what may cause the not so good confirmations, hare are a few things I trying now:

1) Instead of using model generated by Run Model straightly right after Fit Definitive Screening, I used Stepwise Regression to pick terms using AICc and BIC criteria. Then I fit the selected terms with Standard Least Squares. I got some improvement.

2. I can also fit the above selected terms with addition runs assuming the model is underfit. Question: how do decided if a model is underfit or overfit? What is the solution is it is overfit?

3. Should I even try nonlinear regression fit?

Thanks.

Mark_Bailey

I understand that your system is a computer simulation. You are designing a computer experiment, presumably to fit a surrogate. Your simulation does not include any stochastic element, so the same input values lead to the same outputs. You are not screening inputs, for which the DSD is intended. You know the inputs because you can examine the simulation. I presume that the simulation is impractical (e.g., computation takes to long for smooth graphics) so you want to use a good surrogate.

The DSD and several other design platforms are intended for physical experiments in which the response includes stochastic components. You should have tried the space-filling designs. They are meant to be used with computer experiments. Also, they support fitting the Gaussian Process model, which is typically a much better surrogate than the linear regression model.

So I think that the bias in the linear regression model (i.e., lack of confirmation) is because of a non-linearity in the response that cannot be modeled well with a polynomial function.

If you combine the original DSD results with the new confirmation results and fit the model again, does the fit and the predicted response improve?

ZenCar

Mark,

There are many factors can affect response we want to control. We want to identify which are the main contributors. Through engineering experience, we were able to narrow it down to eight factors. The purpose of the DOE is to further narrow it down and get practical direction on how to improve the response. For this purpose, I think DSD does an excellent job.

I have tried two things:

1) Within the DSD platform, I removed a few not as significant terms and trimmed the model from 5 MEs, 4 quadratics and 1 interaction down to 4 MEs, 1 quadratic and 1 interaction. I was able to get more desirable confirmation results.

2) I simply add 4 of the 5 confirmation runs to the original DSD table, and use JMP Analyze: Fit Model : Standard Least Square to the same set up as 1) (4 MEs, 1 quadratic and 1 interaction). I was also able to get more desirable confirmation results with the remaining confirmation run data.

Question: Can I say screening effective factors and generating predictive model are two things, i.e., the later may be due to other reasons such as not having enough data (runs) to fit the model, however, the screening results are still sound and applicable?

Your point about space-filling design is very interesting. I will look into it and circle back in a few days.

Thanks and Happy Holidays!

P_Bartell

OK...I can't resist. I've been sitting on the sidelines on this thread...but now I can't help myself. @ZenCar Your question in your last post is at the heart of this narrative. DSD are first and foremost screening designs. DOE for predictive purposes is something completely different. One of my pet peeves of DOE practitioners is they start with a practical problem that is fundamentally an optimization goal for a set of responses...and then somehow forget the tried and true method of sequential experimental through inductive reasoning. George Box and others have discussed this for years. It's the way to proceed. That's at the heart of the difference between all screening designs and other designs more adept at prediction.

Also to @statman 's earlier points about 'noise'. Noise in future operation of a system comes in all manner of forms shapes and sizes. Many can't be built into designed experiments. For example, when I worked in industry, often production systems would be shut down for major maintenance and capital upgrades. The rule of thumb was 'You never want to be first up on the coating machine after a capital shutdown. The magic of physics and chemistry you thought you understood may not now apply based on the process changes that have occurred.' How you gonna build a major capital systems upgrade into a DOE before the upgrade happens? Not practical and can't be done.

So if predictive power of DOE based models falls apart in the future...why is anybody surprised? Just means more work is needed to fully understand the process.

Rant over. Thanks for reading this far if you have.

ZenCar

Hi @P_Bartell ,

Your comments are quite enlightening. I really appreciate you pointing out that DSDs are first and foremost screening designs. Yes, that was the primary goal before people want to get more out of it.

Coupling with @Mark_Bailey point about the determination of simulation software, I am going to question about the need of accurate predictive model while we now know the direction from DSD and can get exactly answer from the simulation software.

Thanks for such a pleasant writing. Enjoy your holidays.