Which Model Outputs to look at? (Standard Least Squares)

wendytseng

I am preparing for a customer meeting and am wondering if someone has already done this in the past...

 

They would like to know which outputs to look in the Fit Least Squares output when analyzing an experiment.  Does anyone already have something put together with screenshots and a brief description of what to look for?  

 

Thanks in advance.

Laura_Higgins

I don't have anything put together, but I emphasize the following (I pick Screening for personality for a DOE analysis presentation too):

 

- Look at the actual by predicted plot. Are there clumps of data points off the red line? These are being poorly fit by the model, select them and investigate in graph builder. Are the data all on the blue line? All your model explains is the mean. Do the points lay on the red line at the top and bottom? If not, your model predicts poorly where the points leave the line. Frequently models don't perform well across the whole data range and the user needs to think about if this matters. For example, do they care the model performs poorly for out of spec measures versus in spec?

 

- Overall significance - don't throw the baby out with the bathwater approach if it's p=.1, etc. you still have signal from a DOE, but proceed with caution. Especially if you spent a lot of money. You can also partition results to understand a non-significant DOE.

 

- R-square and adjusted r-square = you want them high and together (max = 1, min = 0). If adjusted r-sq is close to 0 or negative and far away from R-squared, your model is too complex for the number of runs and you need to take something out of your model. R-Square = This is how much variation is explained about your Y variable by the model. Low r-square means your factors and factor settings don't have high impact on the outcome, or there is a lot of noise to the signal (did they do replicate runs to check their system)? Also just a few runs don't have a lot of power if there is much noise. I don't like that the Summary of Fit doesn't appear in Screening, so turn it on or use Minimal Report personality and turn a few other things on. Adjusted R-square is really common in DOE to drop well below the R-square with marginal results and people are SURE they can test every interaction with minimal runs.

 

- Lack of Fit - if this is on, add more to your model. You can show this with the popcorn data really easily - main effects only turns it on. Details on this rarely matter to customers, it's simply a sign you need to add interactions or higher order polynomials to the model, hence something is missing/lacking. I've seen this inform people to add tertiary effects with gasses, etc. Something they weren't even looking for but made sense.

 

- Look at the Residual Plot. If it looks weird (smiles, frowns, cones, then investigate the data in graph builder. Perhaps fixed by Y-transformation. You can mention box-cox but that's in the weeds. 

 

- Scaled Estimates (in this order!) - Getting them to understand the picture is easier to do the next 2 tables - this is SO helpful - sort by the scaled estimate and it's the Pareto of up and down effects in the DIRECTION of the Y variable! Get them to focus on the grey bars in the middle. Screen size 3 Increases dissolution, while screen size 5 Decreases dissolution, etc. Then if they want more detail the next 2 sections, but you can now keep them brief because they have an idea of what's happening for the results, which is really what they care about. Stat test here is same as Parameter Estimates table.

 

- Parameter Estimates (after scaled effects)- the coefficients in the model they built. I show them the prediction expression with this. That t-test is checking if it overlaps with 0 (zero).  

 

- Effect test - was each factor significant? The Effect Test is the same as a Parameter Estimate in some cases (continuous variables) but is testing across levels for nominal/ordinal data. The way nominal  levels are coded one is not reported. You can adjust this with the value order column property (last one is left out). Easier to interpret too when you have a categorical variable interaction with a continuous one (screen size x mill time for example) the way these get written in parameter estimates is implicit difference between one of the categories so one is left out.

 

- Then the profiler! I usually start with assessing variable importance, and I don't see people doing that. It helps your presentation because things are now in the right order of important to unimportant, and unimportant stuff is far away from the Y axis. You can also color it and it emphasizes what you just went through "look how important Screen size is! - no need to go deep here, it's a visual tool. Then optimization, and simulation. 

 

- One note for doing statistical tests among nominal levels (like screen size) - use the leverage plots as all the tools are there.