How many data points for a reduced order model?I'm once again in the situation where I think I know what I want to do, but can't work out how to formulate the question correctly... Perhaps someone would be kind enough to help a struggling engineer?
I'm trying to build a reduced order model - I have a complex, mechanistic model and want to develop a simple correlation to describe performance of my system in a limited operational space. I've used JMP to design a "Space Filling" experiment and written a script to run my mechanistic model to fill the results table. I can import the results into JMP, and have been successful in building correlations that meet my needs with the "Fit Model" platform.
However, the mechanistic model is relatively slow-running, so I'm motivated to try to work out how few mechanistic runs I can get away with but still generate a good general fit. It feels like a should be able to use my existing data set to start to answer that - I can manually mask the data, fit to a subset, and then challenge that fit against the larger data set, comparing the r2 on the unseen data, for example. Doing it manually is OK, but time consuming (or perhaps I"m just lazy) - is there some magic widget that I'm missing?
I guess I could use the "Neural" platform and increase the "Holdback Portion" until the Validation set's r2 starts to climb. I'd prefer to stick with a "linear" model (albeit with quadratic and cross terms), but I guess that would estimate the limit. It also looks like I can only pick the Holdback once, so I have to relaunch the platform to play that game.
Thanks in advance!
Charles
Data AccessData Blending and CleanupDesign of ExperimentsPredictive Modeling and Machine LearningSharing and Communicating ResultsRe: How many data points for a reduced order model?Thanks again, Dan. I understand and agree with your response.
The underlying (mechanistic) model is quite non-linear, so my problem is that I'm not sure ahead of time if a quadratic model will be good enough (or which of the ~10 input parameters might show non-linearity). I like the aNN in fact it's likely to pick up any non-linearity but I dislike that it has so many parameters, as you say.
I didn't realise that the Holdback portion of the aNN was used in the fitting of the parameters directly - I thought it was "only" used to choose when to stop. Certainly, if I increase that holdback fraction far enough, the quality of the model reduces (both fitted and validation). I actually managed to write a cute script that runs the aNN model several times inside a loop with increasing Holdback fractions, and then captures the r2 for the fit and validation sets, so I can get a feel for where the model quality starts to deteriorate...
Anyway, thank you very much for your help - I now know more about the DoE platform and feel better able to tackle this sort of problem in the future! If I know that the model output is likely to be relatively smooth, I will likely use a more structured design like you recommend, and test the edges / center points. If I'm less confident, I will use a space-filling design - using more data points to get a fit, but hopefully understanding the non-linearity of the system better (if its there)
All the best
Charles
Re: How many data points for a reduced order model?The difference between the designed experiment and the neural network is related to the model that is fit.
Neural network models can capture VERY nonlinear behavior. In order to do that you need much more data. But the resulting model is much more complex and "wiggly" than the quadratic model. There are typically MANY parameters in a neural network model. Further, neural networks require a holdout set of data (as you know), which means that not all of the data is used for fitting the model. That also means you need a larger dataset since only a percentage of the data is used to build the model. Note that the holdout sets are used to determine when to stop the neural network parameter estimation though, so holding out larger percentages of the data is not quite the same as using less data. The hold out set is still being used, just to a lesser degree than the training data.
So which approach is best would depend on how nonlinear you expect this mechanistic model to be. It will be a Cost/Benefit or Risk/Reward tradeoff for you.
If you truly believe the quadratic model does an adequate job of estimating the relationship, go with the experimental design approach and less data.
If you expect the neural network does a better job of capturing the relationship, go with the neural network, but at the cost of more data being needed.
Re: How many data points for a reduced order model?Thanks Dan. I worked through this, and the answer came back as 21 cases (instead of the ~2200 that I actually ran). All my "power" values in the Power Analysis came back as 1, so I think this implies that the the model should be good.
With that said, I went through using the Neural platform to fit portions of the original 2200 cases. I set up a script that ran the fit 4 times, and I compared the r2 and q2 as I increased the Holdback fraction. Both were good (> 0.995 for at least 3 of 4 of the models) when my holdback fraction was <0.925. When I held back 0.95 of the variables (ie I used 0.05*2200 = 110 conditions used for fitting), then the r2 was < 0.985 for 3 of 4 fits and the q2 was <0.97 for 3 of 4.
I'm not sure why I need more cases when I use the Neural platform. I guess the "custom design" uses a -1 / +1 pattern where my original space filling design was distributed through the space and there therefore perhaps less insightful?
Thanks
CharlesRe: How many data points for a reduced order model?So you have a quadratic model that you feel comfortable with the fit. Now you want to determine how many runs would you have REALLY needed to obtain a similar fit, right? Are you more interested in the predicted value from the linear model or in estimating the parameters of that linear model? I will assume you are interested in the prediction.
So maybe this approach would work. Choose DOE > Custom Design. Enter your factors. Specify the RSM model (main effects, all 2-way interactions, and all squared terms -- I think that is what you said you have). Look at the number of Default runs. Click Make Design.
JMP will create an I-optimal design (best for prediction) with that number of runs.
Open up the Power Analysis portion of the output. Fill in the Anticipated RMSE (this would be the RMSE from the model you have already fit). Enter the Anticipated Coefficients from the linear model you already have. Click Apply Changes to Anticipated Coefficients. The Power numbers that result are the probability of this design detecting that effect. You would want these numbers to be high. How high will be up to you. If they are not high enough, go back and increase the number of runs in the design.
I would also encourage you to look at the Fraction of Design Space Plot and the Prediction Variance Profile. These graphs will show you the prediction "accuracy" of the model with that number of runs. Look for low prediction variance that is fairly flat throughout the entire design space.
I hope this helps!
