Logistic Regression and Model Reduction - Discrepancy between model fitting resultsI'm using the Generalized Regression platform with Binomial Distribution to fit a Logistic model, and then do model selection (e.g., Best Subset - AIC) to reduce model complexity.
Once the model is reduced, I relaunch it with Only Active Terms, using the same exact platform -- Generalized Regression with Binomial Distribution, so that I should get exactly the same fitted model as the one reduced, with same coefficients and predictions.
Nevertheless, when I compare the two profilers, I obtain slightly different predicted values, and extremely different prediction intervals.
Can you provide an explanation of why this is happening?
Also, which of the two fits should I trust?
Thank you!
Re: Logistic Regression and Model Reduction - Discrepancy between model fitting resultsHi Micol,
The reason for this discrepancy is that the two models are using different estimators for the covariance of the parameter estimates (which affects the standard errors). Whenever we do variable selection (Lasso, FS, …), it uses a sandwich matrix, as described in Zou 2006 and Huber and Ronchetti 2009:
Zou, H. (2006). “The Adaptive Lasso and Its Oracle Properties.” Journal of the American Statistical Association 101:1418–1429.
Huber, P. J., and Ronchetti, E. M. (2009). Robust Statistics. 2nd ed. John Wiley & Sons.
Whenever we do an MLE like in the logistic/binomial fit, it uses the usual Fisher information matrix. For a reduced model you arrived at using stepwise regression and then re-fit, we think it is fine to use the secondary results (with smaller confidence intervals).
I hope this helps!
Thanks,
Adam Morris
JMP Technical Support
