The attached JMP table shows a simulation scenario. The column “p true” has a model for simulating binary outcomes Y which are shown in column “Y cont” in continuous modeling type and in column “Y cat” in categorical modeling type. Having the same outcome in different modelling types was necessary for later analyzing them using different platforms. The model used to simulate Y is a simple intercept plus slope model in logit form: logit(p) =15+60*X.
The “p true” formula is a function of X. The table has 42 — arbitrary number — rows equally spread at -1, 0, and 1 levels of X — i.e. 14 rows at each of the 3 levels.
The scripts saved on this table refer to 3 ways to model Y — namely Gen Regression, Nonlinear Regression, and Nominal Logistic Regression. The column “p formula” was created to satisfy the need for a column formula while doing nonlinear modeling.
My expectation was to see red flags in the report of the 3 different platforms used to model Y in the form of very large confidence intervals (if any). This is because the X design cannot provide unique evidence of the parameters used in the “p true” model — i.e. intercept = 15 and slope = 60.
According to the results from running the saved scripts, I see that the expected red flags for the estimation of Y are obvious for the Nonlinear and Fit Nominal Logistic platforms. The Nonlinear platform cannot generate profile likelihood based CIs and the Fit Nominal Logistic has — besides a warning sign (unstable) — very large CIs. The problem is in the Gen Regression platform. The report for the full model does show inflated CIs. However, if I use model selection, the CIs result extremely narrow almost implying certainty on the point estimates which are largely biased as expected.
Question: how should we interpret this ambiguity btw full model CIs and selected model CIs? What if the selected model form was different than the full model, how would we perceive this issue? Can the Gen Regression provide red flags for complete separation? I used JMP15.1 Pro in all this.
Hi Fabio,
Here are some comments from the Genreg developer:
"There are two things to keep in mind I think.
- The penalty in the lasso or especially the elastic net can help avoid separation. One of the things that can happen with separation is that parameter estimates get unstable, but shrinkage will help stabilize them.
- Variable selection helps with separation, but it also makes inference less trustworthy. I often suggest taking the SEs and confidence intervals produced after doing variable selection (in genreg or elsewhere) with a grain of salt. The idea of doing post selection inference is currently a very active area of research and it’s something I would like to address in the future. But right now, looking at things like tests and intervals after doing selection is kind of iffy. So I wouldn’t put too much stock in the confidence intervals that he’s looking at."
Hope that helps.
Adam
