nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Very High R square and significant lack of fit problem

Ella

Hi,

I made custom design for 7 factors with a model containing main effects, interractions and 2 order powers. Number of sample size is 150. I see that 3 of factors are important and made model for these 3 variables.

However i faced with significant lack of fit problem and 0.99 R^2 value.

I searched a lot about the reasons of this situation but i could not find a clear answer.

Do you have an idea about it, what causes this problem?

Kind regards.

Accepted answers

SaraA

Hi @P_Bartell

If it's prediction across the factor space, how impactful is the magnitude of the prediction error across the experimental space from a PRACTICAL point of view?

Could you explain what you meant here exactly? Did you mean that lack of fit, as a measure of the prediction error of the model, could be of little significance if the purpose of the DOE is to see the effect of the factors on the response across a certain experimental space, contrary to performing a DOE to optimize a reponse?

statman

Just to re-iterate, R-square by itself must be interpreted carefully. R-square will always increase as you add DF's to the model. But the point of creating a model is to include only the useful terms (active and significant). So you should look at the delta between the R-square and the R-square adjusted. As the delta increases, it is an indicator your model is over specified.

Looking at your residuals, there is an indication of a non-linear effect. Your model should consider perhaps adding a quadratic effect as suggested by @modelFit

All comments

ThuongLe

Is it possible to share your data and script?

Dan_Obermiller

A lack of fit test compares the variance from the "model not fitting" to the variance of the replicated points (pure error). A significant result means that the "model not fitting" variance is larger than the pure error. The most likely cause is that the model form you are using is not appropriate for the data.

Consider this picture:

A high RSquare value, yet a significant lack of fit because a straight line does not describe the relationship. Remember that a high RSquare by itself does not mean you have a good model. A high RSquare means you have explained much of the variance in the response, but you still may not have a good model. You must consider other items such as model significance, residual analysis, etc. to determine if you have a good model.

So I would recommend plotting your model residuals to see if there are any patterns. A pattern would indicate what might be wrong with your current model form. Also check to ensure that the residuals follow a normal distribution as that could also lead to a lack of fit situation.

Finally, it is possible to get a significant lack of fit test if your replicated points have an extremely low variance rather than the model form being incorrect. This does not happen that often, but I have seen that occur. In such situations, you should look into determining why or how you achieved such a low variance. That can provide some good insights.

Ella

Thank you very much Dan,

I exactly facing all problems you have stated .

normality problem in error terms, i think have some trend in my error term (if you have diffrent opinion please share with me) and very very small pure error variation.

Capture 1.PNG Capture 3.GIF Capture2 .GIF

If I redesign experiment and include cubic term of factor, do i solve this problem?

(By the way, i know my dependent variable depends on other factors that are not included in model but i set them constant when i make DOE and run experiments. I guess this does not cause this problem, right?)

P_Bartell

I concur with all that @Dan_Obermiller contributed. From your residual plot it looks like the root cause of the lack of fit is a relatively small replicate response variance compared to the total response variance.

Here's another thought for you to consider...what is the purpose of experiment/study? If it's prediction across the factor space, how impactful is the magnitude of the prediction error across the experimental space from a PRACTICAL point of view? If it makes little practical difference, then, quite frankly, I couldn't care less that you have LOF.

Ella

Exactly reason of LOF is very small variance of replicates as also seen in pure error value.

Aim of this study is to model dependent variable and make optimization to find optimum Factor levels which return dep. Variable is in between 0.35 and 0.36.
If you look at my current model error values, they are changing between -0.3 and 0.3 which is very big for my target. That’s why, i am trying to find better model explaining my dependent variable to make a better forcast.

SaraA

Hi @P_Bartell

If it's prediction across the factor space, how impactful is the magnitude of the prediction error across the experimental space from a PRACTICAL point of view?

modelFit

@Ella It seems to me that you have unmodeled mean structure remaining in the residuals. By the looks of it you fit f(X) = a + bX + e and by observation, the model f(X) = a + b1X + b2X^2 + e may provide a better fit? What do you residual look like if you run the latter model?

statman

Looking at your residuals, there is an indication of a non-linear effect. Your model should consider perhaps adding a quadratic effect as suggested by @modelFit