Hello, everyone
I am Yusuke Ono, a tester in JMP Japan.
It seems to me that when k-fold validation is used, the predicted values sometimes become very noisy. Please see the screen shot below. This is a plot of predicted values for a model with one X variable. The 5-fold validation is used.
I would like to suggest that we predicted values be calculated simply as below.
(0 + :Name( "y Predictor FoldA==1" ) + :Name( "y Predictor FoldA==2" ) +
:Name( "y Predictor FoldA==3" ) + :Name( "y Predictor FoldA==4" ) +
:Name( "y Predictor FoldA==5" )) / 5
The current prediction formula is as below.
If( :N Val > 0,
(0 + :Name( "FoldA==1" ) * :Name( "y Predictor FoldA==1" ) + :Name( "FoldA==2" )
* :Name( "y Predictor FoldA==2" ) + :Name( "FoldA==3" ) *
:Name( "y Predictor FoldA==3" ) + :Name( "FoldA==4" ) *
:Name( "y Predictor FoldA==4" ) + :Name( "FoldA==5" ) *
:Name( "y Predictor FoldA==5" )) / :N Val,
(0 + (1 - :Name( "FoldA==1" )) * :Name( "y Predictor FoldA==1" ) + (1 -
:Name( "FoldA==2" )) * :Name( "y Predictor FoldA==2" ) + (1 -
:Name( "FoldA==3" )) * :Name( "y Predictor FoldA==3" ) + (1 -
:Name( "FoldA==4" )) * :Name( "y Predictor FoldA==4" ) + (1 -
:Name( "FoldA==5" )) * :Name( "y Predictor FoldA==5" )) / :N Trn
);
Could you please tell me your opinion?
Steps to Reproduce this phenomenon
(1) Run the script below to create a sample data set.
Names Default To Here(1);
Random Reset(111111111);
n = 10000;
x = J(n, 1, Random Uniform(-10, 10));
y = Sin(x) + J(n, 1, Random Normal());
As Table(x||y,<<COLUMN names=""></COLUMN>
(2) Select [Add-Ins] > [XGBoost] > [Make K-Fold Columns].
2-1. Type "1" for [Number of K-Fold Columns To Create].
2-2. Specify "y" to [Y, Response].
2-3. Click [Go].
(3) Run the script below.
XGBoost(
Y( :y ), X( :x ),
Validation( :FoldA ),
Fit( max_depth( 3 ), alpha( 10 ), lambda( 10 ), iterations( 50 ) ),
);
(4) Select [Save Prediction Formula] at the red triangle menu at the "Model 1" outline.
(5) Run the script below to see the saved predicted values.
Graph Builder(
Size( 1209, 478 ),
Show Control Panel( 0 ),
Variables( X( :x ), Y( :y Predictor ) ),
Elements( Line( X, Y, Legend( 5 ) ) ),
SendToReport(
Dispatch( {}, "x", ScaleBox,
{Format( "Fixed Dec", 12, 1 ), Min( -3 ), Max( 3 ), Inc( 1 ),
Minor Ticks( 1 )}
)
)
);
(6) Run the script below to see the plot of my suggestion.
New Column( "My Suggestion",
Formula(
(:Name( "y Predictor FoldA==1" ) +
:Name( "y Predictor FoldA==2" ) +
:Name( "y Predictor FoldA==3" ) +
:Name( "y Predictor FoldA==4" ) +
:Name( "y Predictor FoldA==5" )) / 5
)
);
Graph Builder(
Size( 1209, 478 ),
Show Control Panel( 0 ),
Variables( X( :x ), Y( :My Suggestion ) ),
Elements( Line( X, Y, Legend( 5 ) ) ),
SendToReport(
Dispatch( {}, "x", ScaleBox,
{Format( "Fixed Dec", 12, 1 ), Min( -3 ), Max( 3 ), Inc( 1 ),
Minor Ticks( 1 )}
)
)
);
The attached data table has these results. Please run the table scripts to see the plots.
Thank you for your opinion in advance.
These are good questions to ask Yusuke. If we break down the overall process into two main steps, first is determining one or more reasonable XGBoost models. For this k-fold cross-validation or something similar is invaluable. Second is making predictions, and let's divide this into making predictions on the current training data and on new test data. For predictions on the training data, using any model that was trained on the same data will result in predictions that are typically overfitted and unrealistically good, so out-of-fold predictions make the most sense. For new test data, averaging predictions from all models that have not seen that data is good as you have suggested. (The add-in in fact does this already for all rows for which Y is missing.) It is also okay to perform a subsequent final fit to the full data with no validation, using parameters determined by cross-validation. My experience has been that the predictions from averaging k-fold ("bagging") and those from a full-data fit tend to be very similar, although you could even average them, for example 0.5 * (bagged k-fold predictions) + 0.5 * (predictions from full data fit). I don't think trying to find the "best" among the k-fold model fits is worth trying, since they are operationally equivalent and you run the risk of overfitting with such a choice.
Thank you for your reply, Russ.
I now understand that these out-of-fold predictions by JMP's XGBoost can be used for evaluating models, detecting outliers and making ensemble models.
I think they are several ways to create final predictive models when we use k cross validation.
- Refit a model with selected tuning parameters for all data. (< This is your advice.)
- Average k models. (< This is my suggestion.)
- Use only the "best" model among k models.
Do you think the first method is usually better than the second and the third one for getting a final predictive model?
Thank you Yusuke for the question and very nice description of how to reproduce it with JSL. You have hit on an important difference between the way the add-in works when computing predicted values in comparison to other predictive modeling platforms in JMP. The predictions from the add-in are "out-of-fold", that is, each predicted value is computed only from model fits that exclude that observation.
Take basic 5-fold as an example and assume the first row in the data is assigned to fold 3. The add-in fits five models, holding each fold out in turn, but the predicted value for that first row will only be based on the third fit that excludes fold 3. Similar logic is used for all rows. Why do this? Several reasons:
1. The out-of-fold predictions provide a realistic assessment of how the model will perform on future, unseen data. In fact, all output statistics and graphics labeled "Validation" are based on the out-of-fold predictions. As you have noticed, the variability of these predictions can be much larger than predictions computed with "in-fold" data, that is, when the same data are used for training and prediction.
2. The out-of-fold prediction column is a great way to look for outliers and errors the model may have made across the entire training set. K-fold is nice in that each observation is held out exactly once, and thus we have an out-of-fold prediction for every row.
3. The out-of-fold prediction column can be used to ensemble with similar columns created from other good models. This can be done by simple averaging or by using the out-of-fold columns as new predictors in a second-level model in a technique often called "stacking". The second-level model is typically simpler than first-level models, for example, a lasso model from Generalized Regression or even another XGBoost model with shallower depth. Stacked ensemble models tend to have better predictive performance than any of the first-level models.
To obtain the straightforward training-based predictions you suggest, one good way is to fit a new XGBoost model with no Validation column specified. Another related way is to make a copy of the data table, set all of the Y values to missing, and concatenate it to the original data. The predictions in this case for the rows with missing Ys will be the average of predictions across all fitted models, a technique referred to as "bagging".
Lastly, wanted to mention that the complex formula you display is a result of covering all cases. For example, if a row in the original data has a missing Y, then we want to use the bagged prediction over all models for it.
