Leave-one-out cross validationCan someone help me understand the graph that is shown below from Gen Reg's platform for leave-one-out validation (or perhaps help me better understand leave-one-out validation)? I used Diabetes and am predicting Y (continuous). Why is one particular record being shown in the validation plot? It was my understanding that the solution path Validation Plot plots average value across all leave-one-out's.
Data set attached below with script.
Re: Leave-one-out cross validationHi Wendy,
I think if you look at the help documentation for k-fold it will help to explain this. Leave one out is just a special case of k-fold where k = number of rows of data.
"To obtain the final model, all k models derived for the optimal value of the tuning parameter are fit to the entire data set. Of these, the model that has the highest validation log-likelihood is selected as the final model. The training set used for that final model is designated as the Training set and the holdout fold for that model is the Validation set. These are the Training and Validation sets used in plots and in the reported results for the final solution."
So you are only seeing the plots for the one of the k models: the model with the highest validation log-likelihood.
I agree it is a bit weird to just see the actual vs predicted for a validation set with one data point. But hopefully this explains why now.
Cheers,
Phil
Re: Leave-one-out cross validationHi, Wendy,
Yes, Scaled -Loglikelihood (=negative log-likelihood divided by # observations) used in K-Fold or Leave-One-Out validation method is averaged, but the diagnostic plots such as Actual by Predicted show the results for a single model.
The solution to the Lasso penalized regression depends on a single tuning parameter. When searching for the optimal value over the grid using LOO validation, there will be N (e.g., 442 in your case) models fit at each value of turning parameter. A lot of models will be fit! The scaled -LL values will be calculated for each of 442 LOO observations, and an average of them is then compared to another average scaled -LL from models fit with a different turning parameter value. This is what is shown in the Validation Plot. (Note that the values of the Magnitude of the Scaled Parameter Estimates are equivalent to the values of the tuning parameter.)
Then a final model is selected out of 442 models as described in the documentation from @phil_kay : it is the model showing the highest LL on the validation from "all k models derived for the optimal value of the tuning parameter are fit to the entire data set." The Actual by Predicted plots the final model's results on the training set and LOO observation; in this case, it is Row 305 as this data point yields the highest LL value when used as the LOO validation.
As you can see the K-fold and LOO validation methods operate a bit differently in penalized regression than in Partition and NN. I hope this helps.
Jian
