removing terms from a model following a designed experment

Question

Ok, suppose I've conducted a full factorial design with four 2-level factors. When fitting the model I've included all 2-way interactions. Some of the factors' main effects and interactions are not significant at 0.05. If I then remove the least significant factors (those at the bottom of the effect summary table), some of the non-significant terms become significant. If I were trying to develop the 'best' model, I would feel comfortable doing this. But if I am trying to determine whether a factor plays a role in the response variable then I'm not sure this is legit. Is there a proper rationale, in a DOE context, for removing insignificant terms and then describing the remaining terms as significant?

gchesterton · Answer

The experiment was "open sourced" where volunteer participant teams conducted four runs (randomly assigned treatments) from the master design matrix. I blocked for the team effect, as well as other nuisance factors such as the scenario (of which there were four). Even without these nuisance factors, I expected a team's score (the response variable) to vary despite the factors' effects. These nuisance factors were an unfortunate compromise between "realism" with human subjects and a more tightly controlled experiment.

statman · Answer

Why 4 replicates?  What was changing within and between replicate?  Does the RMSE look reasonable and comparable to the variation you see in the process?  Is the process stable?

The commonest of defects in DOE are (paraphrased from Daniel):

* Oversaturation: too many effects for the number of treatments
* Overconservativeness: too many observations for the desired estimates
* Failure to study the data for bad values
* Failure to take into account all of the aliasing
* Imprecision due to misunderstanding the error variance.

Point 5 is important, particularly if you are relying on p-values for assessing significance.

Victor's advice regarding use of other statistics to determine an appropriate model are worth consideration as well.

Victor_G · Answer

To add some remarks and comments in addition of @statman answer.

Your topic seems very similar to models evaluation and selection. In this process, statistical significance may be one metric, but not the only one (always add domain expertise !). 
Complementary model's estimation and evaluation metrics, like log-likelihood, information criteria (AICc, BIC) or model's metrics (explanative power through R2 and R2 adjusted, predictive power through MSE/RMSE/RASE, ...) offer different perspective and may highlight different models.
You can then select based on domain expertise and statistical evaluation which one(s) is/are the most appropriate/relevant for your topic, and choose to estimate individual predictions with the different models (to see how/where they differ), and/or to use a combined model to average out the prediction errors.

"All models are wrong, but some are useful" is a quite adequate quote when dealing with multiple possible models and trying to evaluate and select some to answer a question.

Hope this additional answer may help you,