Holistic (or Data Driven DOE)<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />We’ve created a holistic DOE demo for a JMP day and would appreciate feedback and edits for improvement. Primary goal is to illustrate the value of JMP Pro using bootstrap forest vs JMP using imputation and stepwise selection methods for variable selection to aid design of next experiment. This builds upon a couple of demos Phil Kay developed recently. Please unzip the attached zip file and open the journal Holistic DOE Case Study. Open the Messy Production Data outline to see the flow of holistic DOE demo:
- Statistical Modelling: Mixture of EDA, Statistics and Experience
- Messy data complicates the issue and the outline Problem of Correlated Xs presents a dataset created by Phil with one Y and two highly correlated Xs showing that traditional stepwise methods select one X vs Bootstrap Forest selecting both Xs as potentially impacting the Y.
- Holistic DOE outline then builds upon the first two outlines with anonymised messy data from the banding example. It is a random sample of rows with anonymised columns of the banding example. It is a sample of just 20 rows to try and mimic the size of data that might be typical in R&D. After clicking on first data link the rest of the links give the demo flow to illustrate the extent of missing data, correlation of continous Xs, then compare X variables selected by stepwise regression, stepwise regression with imputation, decision tree with informative missing which is also a stepwise variable selection method, and bootstrap forest with informative missing. Bootstrap forest with informative missing gives a larger list of potential Xs that may impact Y and therefore gives greater assurance that we do not miss an important X. At this point it may be worth summarising AB Agri story (without assigning problem/solution to AB Agri): A leading developer and producer of animal feedstuff wasted a R&D cycle by missing an important X when selecting from a set of correlated Xs by using traditional stepwise selection methods. This meant they had to repeat a R&D cycle at considerable expense, were six months late to market which resulted in lost sales and reduced market opportunity. They now avoid repeating this problem by using Bootstrap forest and related variable selection methods that identify all potential Xs from a set of correlated Xs irrespective of whether or not some of your data cells are missing.
- DSD sub-outline then performs a DSD on the six variables selected by bootstrap forest and via stepwise selection methods (which is now valid due to zero or near zero correlation structure of the model effects) results in a model involving 5 of the 6 factors. If we had used stepwise variable selection methods (with or without missing imputation) we would have missed 4 of the 5 important Xs and would have required further cycles of development adding to the time to market, cost of development and reducing the opportunity to sell product early when often a premium can be charged.
It would be great to get feedback for improving the demo wrt brevity, simplicity, compellingness …
Malcolm
Re: Holistic (or Data Driven DOE)This is my latest version of the Holistic DoE (or Data Driven DoE / 3D / D-cubed) pigment demo. The original was a bit of a fudge whereas, in this version, the model derived from the DoE should fit well to the historical data. The historical data was selected to have a high degree of multi-colinearity, which results in stepwise modelling missing important effects.
Please ask if you have questions around the back story to the case study.
Cheers,
Phil
Re: Holistic (or Data Driven DOE)Sorry for the delay in posting the recording below. Great session Malcolm and Phil. Thanks.
Chuck Boiler
______________________________________________________________________
Your recording is now available on the WebEx service site. Click the link below
to play it:
https://sasmeetings.webex.com/sasmeetings/lsr.php?RCID=8d822e09f2cc4a98a0300b360e65f3bd
Monthly overview for JMP SE's-20140908 1655-1
Monday, September 8, 2014 9:55 am San Francisco Time
1 hr 10 min 5 sec
* You can forward this message to others to allow them to play back the
recording. *
-------------------------------------
Additional Options
-------------------------------------
To edit recording information and playback control options, click the link
below:
https://sasmeetings.webex.com/sasmeetings/editrd.php?RCID=aa8a3762ac4e4141847ae0125251b5d9
To view more options for this recording, such as downloading, click the link
below:
https://sasmeetings.webex.com/sasmeetings/viewrd.php?RCID=9667b157b1084061a492381aa18bab76
To view all your WebEx Meeting Center recordings, click the link below:
https://sasmeetings.webex.com/sasmeetings/servicerds.php?SP=MC
http://www.webex.com
Re: Holistic (or Data Driven DOE)
Hi,
I've attached a journal with just the pigment milling case study. All the data tables are coded in script so you shouldn't need any additional files. I will continue to refine this and would welcome improvement suggestions.
Trialled this on a customer today and they seemed very convinced.
Cheers,
Phil
Re: Holistic (or Data Driven DOE)Thanks for the presentation yesterday, guys. I think this journal has a lot of good potential. It's simple and powerful at the same time and shows the value of Pro well, even on these smaller data sets.
Can you share Phil's example when it is ready for comments and suggestions? I'd like to experiment with those data as well.
Re: Holistic (or Data Driven DOE)I think that is doable within 45 minutes. We will see today . I wont be there at the SE Meeting, but let me know how it goes.
Re: Holistic (or Data Driven DOE)We've simplified these materials a little - simpler DSD outcome and added text throughout the journal to guide discussions, in particular the following conversational topics that might motivate consideration of Pro:
- A mixture of EDA, statistical methods and experience is an effective combination, but does it scale?
- What are the risks associated with variable selection from correlated X's to select the X's for the next experiment? What about when your correlated X's contain categorical variables?
- How do you extract useful information to aid variable selection when missing cells result in sparse data tables?
- The effectiveness of DOE depends on selecting the correct set of Xs and range over which to vary your Xs. How do you minimise the risk of selecting the wrong variables and wrong factor ranges when designing the next experiment?
Malcolm
Re: Holistic (or Data Driven DOE)Ok,
Here is what I got for a shorted version that is 45 minutes.
- It maybe fine to use slides 14 - 19 with the title slide to warm things up and wrap it up.
- You may not have time for Mouldmat EDA example.
- Ascombe Quartet is good for improper fit and good warm up to over-fitting example. I think you would have time for this and fits in with the EDA ideology of visual and statistical outputs of JMP/approach to data analysis, but depends on time. Maybe skip this and do brief Mouldmat example? I prefer ascombe to illustrate a few points about making/fitting a model
- Definitely do over-fitting example.
- Definitely do Correlated X's (also introduces Pro)
- Limit Missing data examples for Holistic DOE Example part to only one output/visual. First one is fine. I hid the first two created tables in the script so only the graph builder is shown. I think this is a compelling visual by itself and shows how much missing data there is
- For Holistic DOE Example, Maybe skip the middle parts about the imputed data set except to talk about another way to deal with the missing data, but it does not help, same problems as before (Fails to converge/step-halving limit,etc, only a couple of factors selected and only one is really significant of the ones chosen). Then show Decision Tree with informative missing, and Bootstrap. Using the Variable Selection Summary I think says it all and allows you to skip showing the imputed data analysis. Show the data table with the imputation for an example. If you do have time, then show a quick stepwise to demo problem is same.
- I think your simplified DSD example will work and be simplistic enough.
- I attached a trimmed down PPT and journal with files you sent to me (and the updated DSD completed worksheet).
I hope that helps with streamlining some of the demo.
Chris
Re: Holistic (or Data Driven DOE)Thanks Malcom,
I am combing through it now with the goal of 45 min max and inclusion of stated goals. I would remove the PPT part. Correlated X's part is a good demo of what can go wrong so that should stay in. I will continue to work on it and get back to you.
Chris
Re: Holistic (or Data Driven DOE)Hi Chris,
Sorry for slow response, I've been on the road. There is clearly too much material here for a short effective demo and primarily I'm looking for suggestions for reducing it to a 30 to 45 min demo to try and open eyes of existing JMP users working in R&D to the potential of Pro for variabe selection as an input to designing next experiment. I didn't have time to document demo flow, but you'll get some of the main ideas from the links to Mastering JMP recordings. I'm assuming target audience may not be aware of the potential issues of:
If one or more of these problems arise then unless great care is taken in variable selection, a smaller subset of X's may be selected for further experimentation with the consequence of longer total development time which can have a big monetary impact. Part of the goal is to try and get conversations away from the incremetal functionality of Pro to the incremental value of Pro.
Goal is to try and demonstrate these issues in a demo of maximum duration 45 minutes. So quite a bit of work needed to trim down the current demo to achieve this goal.
Correlated Xs demo was created by Phil and he should take credit or that.
Malcolm
Re: Holistic (or Data Driven DOE)Hi Malcolm,
I am taking a look at it now. Four questions and two comments so far:
Questions:
- What is time frame/total time for the whole session?
- Do you have a rough outline of what you are covering when, for how long, and any breaks? (helps with brevity fine tuning)
- Are you planning on going through the whole power point to set the stage of workshop/day?
- How much time are you planning to spend on the Mouldmat Example and what are the key points you will be making?
Comments:
- I see the Ansombe Quartet example in the Journal to explain a bit of fitting issues. I notice that this data set also exists in the sample data but with a different structure. I personally like the one in the sample data set for brevity of the problem description but also emphasize the graph and statistics are needed to properly see the problem, but I guess it depends on the point being made and how quickly you want to get there.
- I like the simplicity of the Correlated X's example
Will look at the rest and get back to you.
