nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Why is my lack of fit p-value very small?

Chemist9

Hello, JMP community. I have recently finished my experiments following an RSM design. I have inputted the data in the software and ran the model (accounting for interactions and RS). I have excluded the factors and interactions that were irrelevant based on the Effect Summary table. However, the lack of fit p-value remains very small (see screenshot below). I'm afraid the model is overfitting because while this is the case, my RSq value is high at 0.97-0.98 (I also know for a fact that this may not be a good indicator since there are nonlinear interactions among my factors).

I have also tried the Stepwise modelling, and the lack of fit p-value did not improve. May I ask for help on what to do from here or what I should try to find the model that fits my data? I'm stuck at the moment and can't find the optimal conditions I'm looking for. Thank you.

Screenshot 2026-03-23 100416.png

Accepted answers

All comments

Victor_G

Hi @Chemist9,

Welcome in the Community !

The Lack of Fit test is helpful to evaluate if your model fits the data well.
It's very difficult to help you only with a screenshot of this model report. Can you share more screenshots from the different part of the report (actual by predicted plot, residuals, etc...) or even better, the anonymized dataset ? A statistically significant p-value for this test indicates that the model error is largely bigger than the pure error, and that your model may not be adequate.

This situation may happen in your example for several reasons:

If your model is "too simple" for the results/data you have collected, for example if there are some missing terms to model curvature in your data (through 2nd order terms, interaction, quadratic effects, or higher order effects). Based on which info did you remove some terms from the model (p-values only) ? What are the behaviour and performances of the full assumed model ? Since you have done an RSM model, where the target is optimization and prediction, I would recommend using several metrics to refine your model, such as model RMSE (precision of the model) and an information criterion (AICc/BIC) to help you evaluate and balance the complexity vs. accuracy trade-off of your model.
If your replicated points have extremely low variance compared to the model. Are your replicates experiments independent from the other experiments ? How did you run these experiments ? Are they "true" replicates : you have created AND measured these experiments independently (not just measured a second/third time the same experiment) ?

I would also recommend to check the adequacy of your regression model through residuals analysis, to check if regression assumptions are met and if any patterns are detected in the residuals that could indicate model's inadequacy. If a pattern is visible, that might confirm that either some terms are missing from the model, or that a transformation may be needed to fit your data correctly.

Hope this answer will help you,

Chemist9

Thank you for your helpful response, @Victor_G !

To give you a preview of what I have already explored so far:

I tried running the model where all possible two-way interactions and quadratic terms were included. I then excluded some terms based on their p-values indicated in the Effect summary table. I removed terms with p-value > 0.05 (except for Temperature since it has other important interactions with other factors).
For the model where I removed the "insignificant" interactions, I got a normalized RMSE of 5.66%, which I believe is still acceptable for metallurgical processes that I am currently exploring. But let me know if we have a different standard on gauging a good or bad RMSE.
I have not yet explored the information criterion (AICc/BIC) you mentioned. Can I get this from modelling using the Stepwise personality?
Yes, my replicates were performed and measured independently. Is it possible that the model is already nitpicking because of the small variance?

To further aid us in this discussion, here are the relevant plots you mentioned:

Actual by predicted: I personally do not see a pattern (S, etc.) but it looks like most points are concentrated at the upper right region
Residual by predicted: I am unsure if there is a pattern on the plot, but I don't think it is randomly distributed.
Studentized residuals: Based on my current understanding, since all the points lie inside the green and red boundaries, there are no outliers in my data that I need to remove.

My problem now is I do not know where to go from here since I have very limited knowledge on these tools.

As a new member of the community and JMP user, your response was really helpful. Looking forward to our discussion. Thank you!

Victor_G

Ok, this is helpful.
Looking at the actual vs. predicted plot, it seems that some low values may be overestimated, and some high values may be underestimated. I'm also very curious about some "line of points" in these plots: are they replicates, or different points where a term is missing in the model to differentiate them ?

The information criterion is an additional metric available in Fit Least Squares report in the Summary of Fit panel.

Yes, in cases where variance from replicated points is very small, you may get statistically significant lack-of-fit test, with an appropriate model (hence my question about "true" replicates, to make sure you have done them correctly and not underestimate natural experimental variation). You may also have statistically significant lack of fit test more easily with high number of centre points : https://statease.com/blog/importance-center-points-central-composite-designs/#:~:text=The%206%20center%20point%20design,making%20the%20test%20almost%20meaningless.

Please check the Box-cox transformation panel accessible in the Factor Profiling options, to make sure you don't need a small transformation to better fit your data. Percentage values responses usually need a LogitPct or LogisticPct transformation to make sure predicted values stay in the range from 0 to 100. These transformations should help reduce S patterns in the residuals and actual by predicted plots.

Hope this follow-up will help you,

Chemist9

Hi @Victor_G !

I tried doing your recommendations, and I think there has been some improvements in the model. However, I am still unsure if this is the best-fit model for my data.

Those "line of points" you are referring to are true replicates.

I did the LogitPct and LogisticPct, and it seems that LogitPct works better for my set of data. I had a more realistic prediction profiler and a lower value for AICc compared to the LogisticPct. Doing this also limited the maximum Y response to 100%, which I was encountering previously. However, my lack of fit p-value is still very small, although the RMSE improved. Is it possible that I am still getting low LOF p-value because of my center points? I designed my experiments to follow Central composite design-RSM.

I am also thinking of applying a higher-order model. Is there a way that JMP can suggest the possible higher-order interactions? Or is it done in the Fit model window manually?

Screenshot 2026-03-24 085616.png Screenshot 2026-03-24 085622.png Screenshot 2026-03-24 085630.png

Thank you very much for your helpful inputs!