Training and Validation R2s in Lasso/Elastic net

Question

Dear all,

I use k-fold cross-validation in combination with penalized regression models (lasso, elastic net) and want to examine the (averaged) R2 values of the training/validation sets to check for overfitting and explained variance. I already searched around with the many options, but I can not find that. Is there a way to show this? It would be great, if that could be shown in the overview table, so that I can use it for model selection.

Thank you, Markus

scott_allen · Answer

@MarkusJH

You can compare the R2 values across k folds using the Model Comparison platform in the Analyze > Predictive Modeling menu. Follow these steps:

* Create a K Fold column using the Make Validation Column utility and select "Make K Fold Column" (in the bottom left of the dialog window).
* Launch Gen Reg and load your KFold column as the Validation role.
* Run your regression model (Elastic Net, Lasso, etc.) and make sure that Validation Column is selected as the Validation method.
* When you are satisfied with the model, save the Prediction Formula to the data table.
* To view the R2 value for each fold, launch Model Comparison from the Analyze > Predictive Modeling menu
* Load your saved formula column as the Y and the KFold Validation column as the Group
* The resulting report will show you the measures of fit for each Fold in the column:

gemmahodgson · Answer

I asked JMP a similar question about Elastic Net Regression adn how I could get our some form of R-Squared and the reply was that it is not give for KFold or Leave-One-Out validation methods, only for AICc and BIC. So I think you have to calculate it manually yourself. I think it's quite annoying as SAS can do this.

MarkusJH · Answer

... or another thing, one wants to do in the inner loop would be hyperparameter selection, e.g. the number of hidden units.

Of course, at the end of the inner loop, there must be a criterion that defines the best model, which is then submitted to the outer loop cross-validation.