Can I remove a run from analysis?

CYLiaw

I created a resolution IV screening design with four factors (temperature, speed, layer height, and wait-time) and four responses (stress at break, strain at break, modulus, and crystallinity).  The design has 8 runs plus 4 center points, so a total of 12 runs. The goal for my experiment is to identify the most influential factors for each response.  After running the experiment, one of the runs showed a very different result from others (stress at break and strain at break are much larger) since the samples were not printed very well.  I wanted to remove that run (Run 3, Temperature= 410, speed=20, layer height=0.3, wait-time=11) because I can get a much better fit and more factors were detected in the model.  For example, if not removing Run3, the factors which are significant for stress at break are temperature and wait-time. After Run 3 was removed, temperature and wait-time were still detected, and layer height was detected as the third important factor.  Also, for strain at break, temperature is the only factor if Run 3 is included, but without Run 3, temperature and speed were included in the model.  However, when I checked the design evaluation, the prediction variance became higher after removing Run 3, and it's pretty high especially at the factor level of Run 3.  So I am not really sure if it is appropriate to remove that run.  I have included some figures to show the difference between the two designs (with and without Run 3).  Thanks!

Design comparison.pdf

statman

Here are some of my thoughts.

 

There are no definitive "rules" for handling unusual points in experiments, but as Daniel points out, one of the 5 commonest defects in the analysis of DOE is "failure to study the data for bad values".  This is likely due to the fact most experiments have rather small data sets and singular data points can have a huge effect on quantitative methods (e.g., ANOVA and regression).  

 

Now before you start removing data points, I might ask what did you learn from that treatment?  Why were the results so much different than the others. How much different? What were the other variables not specifically manipulated in the experiment (aka. noise) doing.  I remember discussing outlier data points with Dr. Taguchi many years ago and he would say "that may be the mosty important thing learned in the experiment".  Is the measurement system adequate?  For destructive tests such as stress and strain tests where it is virtually impossible to separate measurement error from product variation, this can be challenging to ascertain.  I often recommend multiple samples be made at each treatment so some within treatment assessment can be done.

 

For your situation, first confirm the data is indeed an outlier (since there is a possibility it is due to a high order treatment effect).  Since you have multile Y's, you could test for possible outliers with Mahalanobis Distances (Analyze>Multivariate>Multivariate, option Outlier Analysis). Most folks use residuals to help identify when analysis assumptions are not met. Rank analysis may also be useful.

 

If you are convinced the treatment is indeed compromised, there are a number of things you can do to see the effect of that treatment on results:

Replacement:

1. Replace the unusual data points with the mean of the remaining data points

2. Regress on the remaining data points and using the Save Predited Results, JMP will "predict" the missing data point.

3. Use predicted values (assuming you did predictions before running the experiment)

Rerun:

1. Rerun that treatment, though you may need to account for possible block effects

2. Rerun a fraction of the experiment

3. Rerun the entire experiment, though if it were me I would see what I had learned and modify the levels before replication