nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

NN instead of DOE - any use cases known

martindemel

Hi,

I've got a question from an automotive customer. Sometimes it is not possible to start with an DOE to understand the relationships or the process. This mainly is in cases where there have to be many electrical/magnetic/inter-car simulations to be done. He said they think about using a Neural Net and learn on the incoming data from several sensors instead.I know I have to become more concrete and will talk to him soon about this.

I thought some of you have heard of something like this already (or at best knows more about it).

So any thoughts would be welcome. I'll come back within the next few days with more concrete Information.

Thanks

Martin

Accepted answers

tom_donnelly

Martin,

Thanks for the second post with comments from the customer. When you write, But already before that the idea is to build up a NN without real data, specify the net and start with simulated data based on given ranges one believes could be right. , exactly where do these simulated data come from? Does a computer simulation model of the manufacturing process exist? Or are these “simulated data” really the best guesses/opinions of subject matter experts? Understanding this is important. If a computer simulation model exists, then DOEs can be run along the lines of what Bill was saying he did at P&G. The surrogate model fit to the DOE data generated by the computer simulation could be a Neural Net model or one of the other modeling methods in JMP. I would try them all and pick the best performer.

If the simulated data are being generated by subject matter experts, you could model their opinions using a DOE, but I have never done that and am reluctant to oversell what would be learned. I hope your customer has a computer simulation to generate data. This probably better captures the knowledge of the subject matter experts.

Certainly gathering data from machine and plant interfaces and measurement instruments may prove useful. But, keep in mind if the manufacturers are trying to keep the process running as nearly the same as possible - that is, purposely NOT changing any inputs - then it will be hard to detect any effect for a factor when it is not changed. Historically collected data is most useful when a process shift in the response is detected and then one can check to see if any of the process inputs have changed. There's little knowledge in the same data point being repeatedly collected, other than getting an estimate of the variability.

When you write, If you have processes and/or plants there is often the case you cannot run a DOE at all because you cannot stop the production for a DOE., this in my opinion depends very much on how well the process is already running. If they have an 80% yield they may think they want to improve, but can't afford to stop the process. The 20% scrap/rework is not desired, but historically been acceptable to management. If the yield is 20%, then the economics change dramatically. Shutting down for a week but raising the yield to 80%, in the very short term is painful, but in the not too distant future pays off in a big way. Of course that is easy for me to say...

Dan pointed out that I have several resources on the JMP Federal Government User Resources Page, Just yesterday I finished a series of webcasts for my US Federal Government customers and have posted the recordings there. Below is a table with the links to the topics. On the last day I recorded a session on Space-Filling Designs for Computer Simulations that you and your customer might find useful. These recordings were intended for the customers to refer back to and/or to share with colleagues who missed the session. Please feel free to share/use any of the content that you think will be of value with a customer.

Kind regards,

Tom

Recordings from Mid-Summer Webex Week, July 27-30, 2015

Twelve 25-50 minute webcasts were presented - three per day - Monday through Thursday.

Series 1 focused on Building Better Models (BBM)

Series 2 focused on Design of Experiments (DOE)

Series 3 featured a mix of topics around data visualization and data preparation

The webcasts were recorded and the links are posted in the table below.

The underlined portion of the title in blue is the link.

EDT	CDT	MDT	PDT	SERIES	Mon. July 27	Tue. July 28	Wed. July 29	Thur. July 30
12:00 PM	11:00 AM	10:00 AM	9:00 AM	1 BBM	Building Better Models - Overview and Use of Honest Assessment	Regression - Linear, Logistic, Generalized	Decision Trees - Basic, Bootstrap Forest, Boosted Tree	Neural Networks - Single Layer, Dual Layer, Boosted
1:00 PM	12:00 PM	11:00 AM	10:00 AM	2 DOE	Custom DOE - Make the Design Fit the Problem	Definitive Screening Designs (DSD) - Creation & Augmentation	Covering Arrays - Detect Faults in Software & Systems	Space-Filling Designs for Computer Simulations
2:00 PM	1:00 PM	12:00 PM	11:00 AM	3 Mix of Topics	JMP Tips & Tricks "Wish I knew then what I know now."	Analyzing DSD DOEs - Graphical, Conservative & Aggressive Approaches	Preparing Data for Analysis - Now Easier in JMP 12	Getting the Most from Graph Builder - Tips & Tricks

All comments

Bill_Worley

Hi Martin,

While at P&G we would build surrogate models based on computer simulations that were the "truth". We would use space filling designs for the DOE and give that to the engineers running the simulations. They would return the results and we would use NN or Gaussian models to build the surrogate model. It sounds like your customer has a number of "experiments" already and is using NN to guide understanding.

To me this is analogous to holistic DOE where you use the historical data to find out which factors are most important and then use those factors to guide later DOE development for optimization.

Daniel_Valente

tom.donnelly has a number of really excellent resources on the JMP Federal Government User Resources Pagepecially around this topic -- surrogate models, computer simulation models etc. I'm hoping he will chime in here.

And I agree with billw@jmp -- this becomes a holistic DOE problem. They have a bunch of sensor data -- now what? Let's use some of our data mining techniques to find out the important drivers and the ranges in which to experiment in. Use Bootstrap Forest and cross-validation techniques (or NNs with CV and variable importance). Then find the likely factors to experiment on, and the ranges that are of interest and put together a follow-up DOE to really determine cause and effect.

Sounds like a great JMP Pro angle!

martindemel

Hi,

thanks Bill and Daniel so far. I talked to the customer sooner as I thought I would. So the base is the following and a bit different than I thoght it is, sorry for the confusion:

If you have processes and/or plants there is often the case you cannot run a DOE at all because you cannot stop the production for a DOE. And there is also no laboratory to be used for experiments. So the aim is to gather data from machine or plant Interfaces or from measurement instruments.

But already before that the idea is to build up a NN without real data, specify the net and start with simulated data based on given ranges one believes could be right.

When they start to gather real data this data should be 1. tested with the NN, and 2. the real data should be used to further Train the NN.

So, this case is less a holistic DOE Topic than a modeling and on the run "optimizing the model" Problem. In the cases you mentioned where there is data - yes you would go the path of holistic DOE.

I understand that basically one could use also other models to guide one, but NN are usually the better learner adding new data to the model, becuase it "remembers" the past. Other modeling techniques usually start from scratch with the new data base.

Hope that's a better description now. Do we have an example on a NN getting trained on consistently additional new data?

Thanks,

Martin

tom_donnelly

Martin,

Kind regards,

Tom

Recordings from Mid-Summer Webex Week, July 27-30, 2015

Twelve 25-50 minute webcasts were presented - three per day - Monday through Thursday.

Series 1 focused on Building Better Models (BBM)

Series 2 focused on Design of Experiments (DOE)

Series 3 featured a mix of topics around data visualization and data preparation

The webcasts were recorded and the links are posted in the table below.

The underlined portion of the title in blue is the link.

EDT	CDT	MDT	PDT	SERIES	Mon. July 27	Tue. July 28	Wed. July 29	Thur. July 30
12:00 PM	11:00 AM	10:00 AM	9:00 AM	1 BBM	Building Better Models - Overview and Use of Honest Assessment	Regression - Linear, Logistic, Generalized	Decision Trees - Basic, Bootstrap Forest, Boosted Tree	Neural Networks - Single Layer, Dual Layer, Boosted
1:00 PM	12:00 PM	11:00 AM	10:00 AM	2 DOE	Custom DOE - Make the Design Fit the Problem	Definitive Screening Designs (DSD) - Creation & Augmentation	Covering Arrays - Detect Faults in Software & Systems	Space-Filling Designs for Computer Simulations
2:00 PM	1:00 PM	12:00 PM	11:00 AM	3 Mix of Topics	JMP Tips & Tricks "Wish I knew then what I know now."	Analyzing DSD DOEs - Graphical, Conservative & Aggressive Approaches	Preparing Data for Analysis - Now Easier in JMP 12	Getting the Most from Graph Builder - Tips & Tricks

martindemel

Hi Tom,

thanks for this detailed input. this will help a lot to get to the next discussion and move things further. Also the material is great.

It's correct that the seriousness of saying "no DoE" is dependent on the current Situation. We then Need to talk to the right people who will be able to decide for that.

Thanks again

Martin

malcolm_moore1

Hi Tom,

I particularly liked your session on analysing DSDs and the use of check points to change model selection criterion to max validation R-square with stepwise and generalised regression. Could you please share your data table called CO2 Capture Process.

Thanks,

Malcolm