Which Model Outputs to look at? (Standard Least Squares)I am preparing for a customer meeting and am wondering if someone has already done this in the past...
They would like to know which outputs to look in the Fit Least Squares output when analyzing an experiment. Does anyone already have something put together with screenshots and a brief description of what to look for?
Thanks in advance.
Re: Which Model Outputs to look at? (Standard Least Squares)I don't have anything put together, but I emphasize the following (I pick Screening for personality for a DOE analysis presentation too):
- overall significance - don't throw the baby out with the bathwater approach if it's p=.1, etc. you still have signal from a DOE, but proceed with caution. Especially if you spent a lot of money. You can also partition results to understand a non-significant DOE
- r-square and adjusted r-square = you want them high and together (max = 1, min = 0). If adjusted r-sq is close to 0 or negative, your model is too complex for the number of runs and you need to take something out of your model. R-Square = This is how much variation is explained about your Y variable by the model. Low r-square means your factors and factor settings don't have high impact on the outcome, or there is a lot of noise to the signal (did they do replicate runs to check their system)? Also just a few runs don't have a lot of power if there is much noise. I don't like that the Summary of Fit doesn't appear in Screening, so turn it on or use Minimal Report personality and turn a few other things on. Adjusted R-square is really common in DOE to drop well below the R-square with marginal results.
- Lack of Fit - if this is on, add more to your model. You can show this with the popcorn data really easily - main effects only turns it on. Details on this rarely matter to customers, it's simply a sign you need to add interactions or higher order polynomials to the model, hence something is missing/lacking.
- Look at the residual plot. If it looks weird (smiles, frowns, cones, then investigate the data in graph builder. Perhaps fixed by Y-transformation. You can mention box-cox but that's in the weeds.
- scaled estimates (in this order!) - Getting them to understand the picture is easier to do the next 2 tables - this is SO helpful - sort by the scaled estimate and it's the Pareto of up and down effects in the DIRECTION of the Y variable! Get them to focus on the grey bars in the middle. Screen size 3 Increases dissolution, while screen size 5 decreases dissolution, etc. Then if they want more detail the next 2 sections, but you can now keep them brief because they have an idea of what's happening for the results, which is really what they care about.
- parameter estimates (after scaled effects)- the coefficients in the model they built. I show them the prediction expression with this. That t-test is checking if it overlaps with 0 (zero).
- Effect test - was each factor significant? This is important when there are interactions, etc. and not just the parameter estimates. Easier to interpret too when you have a categorical variable interaction with a continuous one (screen size x mill time for example) the way these get written in parameter estimates is implicit difference between one of the categories so one is left out.
- then the profiler! I usually start with assessing variable importance, and I don't see people doing that. It helps your presentation because things are now in the right order and unimportant stuff is far away from the Y axis. You can also color it and it emphasizes what you just went through "look how important Screen size is! - no need to go deep here, it's a visual tool. Then optimization, and simulation.
Re: Which Model Outputs to look at? (Standard Least Squares)Thanks for the great discussion, Laura.
I have created some slides that others can comment on / edit / re-use.
Re: Which Model Outputs to look at? (Standard Least Squares)This is really nice, @wendytseng . In fact, this week a customer asked for the exact same thing.
I don't think anything like this exists elsewhere in our resources. It might be good to share it more widely, e.g. in a JMPer Cable post.
Thanks,
Phil
Re: Which Model Outputs to look at? (Standard Least Squares)Well said, @phil_kay! I agree. The content in your PowerPoint will make a nice blog post, @wendytseng!
Re: Which Model Outputs to look at? (Standard Least Squares)I know it gets messy, but I would also add on the Lack of Fit slide that you need pure replicates AND an extra level of the factors. I could fully replicate a full factorial design, but if I fit the full model I will not get the lack of fit test.
Re: Which Model Outputs to look at? (Standard Least Squares)@phil_kay and @Jeff_Perkinson,
@Jason_Wiggins and I are going to work on putting something together. If helpful, we can share it and get feedback in an upcoming SE meeting (thoughts @chuck_boiler and @MaryLoveless?).
We are thinking of splitting the outputs into 2 classes:
- Health checks "is my model healthy enough"?
- Interpretation "What statements can I make? What actions can I take?"
With those 2 classes of outputs, putting them in a "menu" framework. Choose which meal you want based on how much time you have / implications of your decisions, the data complexity, and your "temperament" (some engineers and scientists love getting into the weeds and others do not):
- Happy Meal (smallest set of outputs): Just the bare minimum outputs
- Cheeseburger Combo (medium): A good balance
- Big Mac Combo (large or..."Super Sized"): All the key outputs assuming you need to go there given the results and/or have the temperament
To be clear, I am not suggesting that using our software is like eating processed food.
Re: Which Model Outputs to look at? (Standard Least Squares)Great idea, Chuck! Thanks for being one of my editors!
Here is the post in case others are interested. There are 5-parts to it and this is part 1:
https://community.jmp.com/t5/JMP-Blog/An-analyst-s-new-year-s-resolution-Use-models-to-solve-more/ba-p/346098
Last one is to be posted next week on Feb 11th.
