I am currently analyzing a full factorial experiment with 2 levels and 3 factors, unreplicated, n = 8. According to JMP guides, the proper methodology here would be to fit the model using factorial to degree 2, assuming higher order interactions are insignificant, before using backwards selection to reduce the model down to its significant factors.
JMP guide for 2x2x2 full factorial analysis: https://community.jmp.com/t5/Learning-Center/Analyzing-Full-Factorial-Experiments/ta-p/271995
My question is - suppose the data shows that in fact, all main effects and all 2-way interactions are significant. That leaves me with a model with 6 terms, describing a dataset with only 8 observations. At this point I have very high r^2 due to the high number of terms and p < 0.05 for all 6 factors. From an analytical point of view, is this considered overfitting, and would this be considered not best practice?
If so, would a better way to look at this data be to only look at main effects (i.e. looking at the "average" effect of each parameter rather than diving into details)?
Thanks.
