Why the AICc and BIC values under the Std Least Squares personality differ from those under the Mixed Model one?If I use the exact same data and mixed model, I get different AICc and BIC values depending on whether I use the Std Least Squares (SLS) or the Mixed Model personalities. Is there a formula simplification of these criteria for the SLS case? If so, what is it? And what is the case for the Generalized Regression personality?
Re: Why the AICc and BIC values under the Std Least Squares personality differ from those under the Mixed Model one?Time Series and Mixed Model platforms in JMP do not use the same estimation algorithms. ARIMA runs an initialization step to start the Kalman filter, while Mixed Model does not. However, the difference is small and negligible.
Thanks for your questions.
Re: Why the AICc and BIC values under the Std Least Squares personality differ from those under the Mixed Model one?Hi @phersh and @jiancao,
Attached to this message you will find a simulated dataset I put together to exemplify my previous point, the formulas for the simulation are shown.
The response is "y" which is observed through t (time). There is no fixed effect, just an intercept model. The saved scripts show (1) plot t vs. y, (2) ARIMA solution, (3) Mixed Model Solution with AR(1) covariance, (3) Mixed Model Solution with random effect, (4) Mixed Model Solution with AR(1) covariance and random effect. Given the way I simulated the data, solution (4) is the correct one, and in fact is the one with the lowest AICc & BIC.
This is to show what I meant in my previous post, if I want to compare different covariance structures with the same fixed effect specification (here none, just intercept) I am forced to use the Mixed Model personality (i.e. ML). Besides, the AR(1) time series solution (also ML) does not output the same AICc/BIC/-2 LogLikelihood values found with the Mixed Model AR(1) solution - even though I noticed their difference tends to vanish as data size grows which is not shown here.
Re: Why the AICc and BIC values under the Std Least Squares personality differ from those under the Mixed Model one?@fdottaviano good news! You can compare different random effect specification in the mixed model platform.
Thanks again to @jiancao,
You can use AICc or BIC from Mixed Models to compare different random effect specifications with the same or different sets of fixed effects.
As for specification of the covariate structure in repeated measures, I don't know if you can use AICs or BIC to compare that to a random effect model in Mixed Models. Do you have a use case where you would like to compare these? The comparison technique I described above can be used for any model you create.
@jiancao was trying to stress that the REMLs used by SLS would be different for models with different sets of fixed effects even with the same covariance structures (random effects spec is equivalent to CS structure). But I guess this caused confusion:
“The key difference is that in REML LL there is a penalty term, (1/2) log |X`VinvX|, which involves the fixed effects, X, in the model. So, when the fixed-effect part of two models are different (say, main effects vs main effects+interactions with both having same covariance structure), REML LL, thus, AICc and BIC, will be different. This makes model comparison invalid.”
Re: Why the AICc and BIC values under the Std Least Squares personality differ from those under the Mixed Model one?@fdottaviano,
Another great question.
One option to compare ML to SLS or models of all different types of fixed effects and random effects is to save out the prediction formulas from the models under save columns and prediction formula. Then use the model comparison platform under the analyze menu/predictive modeling. This will not give you AICc or BIC partially due to the different modeling platforms calculating log likelihood differently. Another reason for the lack of AICc or BIC is to allow the comparison of tree-based models to linear modeling. The model comparison will give you RSquare, RASE, AAE. It will also allow you to compare residuals for each model
This is only available in JMP Pro.
Re: Why the AICc and BIC values under the Std Least Squares personality differ from those under the Mixed Model one?Thanks @phersh,
Let me see if I got it right.
What I learnt is that "SLS" and "Generalized Regression" personalities both use REML, and the "Mixed Model" personality uses ML.
Since Generalized Regression can only run fixed effects models, there is no risk of invalid comparisons btw models - in AICc or BIC terms -because they are all comprised of fixed effects.
When you want to fit a mixed model (fixed + random effects) you have two alternatives in terms of personalities: SLS and Mixed Model.
If I use SLS, I can compare models - in AICc or BIC terms - with different random effect specifications - e.g. two random effects vs. one random effect vs. no random effect - PROVIDED all models have the same fixed effect specification.
If I use Mixed Model, I can compare models - in AICc or BIC terms - with different fixed effect specifications - e.g. response surface vs. main effect + interactions vs. main effects only - PROVIDED all models have the same random effect specification.
To see how my SAS PROC MIXED training material fits into JMP, we should start with a full fixed effects specification first and test different random effect specifications using REML - i.e. using the SLS personality. Once we select the best random effect specification - e.g. lowest AICc or BIC - we then test different fixed effect specifications using ML - i.e. using the Mixed Model personality. Once we select the best fixed effect specification - e.g. lowest AICc or BIC - we come back to the SLS personality and estimate the final model comprising both the best fixed and random specifications. So far, so good.
However, what if I'm in doubt btw two models with a given fixed effect specification and different covariance structures - e.g. random effect vs. spatial covariance? In JMP I can only use the Mixed Model personality to fit the spatial model. As such, the spatial model can only be fitted via ML which forces me to use the same Mixed Model personality to fit the model with a random effect so they both can be fitted under the same method: which is ML. Now, if I want to select either one based on AICc or BIC wouldn't it be invalid? How could I compare apples with apples in this situation?
Re: Why the AICc and BIC values under the Std Least Squares personality differ from those under the Mixed Model one?Thanks to @jiancao for the answer on this. First off really good question. This comes from different modeling platforms using different Log likelihoods.
The ML LL and REML LL used by Mixed Model and SLS with random effects are shown below @jiancao captured them from SAS documentation https://support.sas.com/documentation/cdl/en/statug/63962/HTML/default/viewer.htm#statug_mixed_sect025.htm)
The key difference is that in REML LL there is a penalty term, (1/2) log |X`VinvX|, which involves the fixed effects, X, in the model. So, when the fixed-effect part of two models are different (say, main effects vs main effects+interactions with both having same covariance structure), REML LL, thus, AICc and BIC, will be different. This makes model comparison invalid.
This is the rationale behind JMP’s decision to use ML LL, not REML. In other words, in Mixed Model personality, you can use AICs or BIC to compare different models with different specifications of fixed effects. (I say JMP’s decision is because SAS Proc Mixed uses REML as the default estimation method but offers ML as an option:cara_que_sonríe_con_ojos_sonrientes:)
An additional note: the "-2 Residual Log Likelihood" in the Mixed Model’s Fit Statistics report follows the REML LL (i.e., includes the penalty term), and matches the -2 Log Likelihood reported in the SLS’s REML Variance Component Estimates section.
For the Generalized regression platform, we do not support random effects at this point, so this would not help in the case.
GenReg has many different techniques such as Lasso and elastic net that allow for variable selection and model fitting in one step. Ridge regression is useful when you have correlated factors. Here is a nice overview.
https://www.jmp.com/support/help/en/15.1/?os=win&source=application#page/jmp/overview-of-the-generalized-regression-personality.shtml
