Efficient tuning of XGBoost parametersHi @russ_wolfinger,
I've been working on building models with XGBoost and comparing them to other tree models and non-tree models, like NN among others. I am doing a similar approach to Nate and holding back a portion of my original data set (in a completely different data table) and then using the model comparison platform to see how well these models compare to each other with the test data set.
I'm finding that XGBoost is not performing as well on the test data set as I would have expected. I find that bootstrap forest does a better job at present. I've compared the default and what follows as the best output from a tuning table for an optimized XGBoost. It could be that I'm not tuning the fit very well, so I'm wondering if there is a more efficient way to tune the fit since there are so many different parameters that one can modify.
I've gone through the examples and read the online documentation referenced in the PDF help, but there's not much description for practical help when it comes to tuning the model. As a note, I am using single hold-out validation and not k-fold cross-validation. Is it better to first build a very complex model that might slightly overfit and then prune it back with the other parameters to make it more conservative?
DS
Re: Efficient tuning of XGBoost parametersHi @russ_wolfinger,
Thanks for the updated information. I did test out as best I could the output from an optimized boosted tree and bootstrap forest against the XGBoost, after adjusting XGBoost close to the other platforms I could and the outputs were well correlated. Not exact, but very close!
Following the procedure in the link I posted earlier has really helped to approach a better tuned model.
On a couple other feedback things:
1. I recently understood what the "Remove shown models" under "Compare" does, which is as it says, but was not clear that if a box was unchecked, it would keep it. This has been a helpful way to eliminate possible fits after sorting the data table by the validation R^2.
2. I have noticed that when running a tuning design, it's important to make sure the first column has the same fixed values as the Min/Max columns or is somewhere in between (if being varied) for it to work properly. There have been a couple times I was tuning another value, e.g. min_child_weight, and had entered in fixed values for the iterations (say Min/Max 150), but the first column value was left at 100. When this is the case, it keeps the iterations at 100, not 150. I would think that if the min/max were set to the same values, the first column should pull that value and re-populate it as it does using the tuning table for the non-fixed parameters. In the example below, I was tuning max_depth and min_child_weight, but had a couple instances where I left iterations at 100, even though the min/max were set to 387. I had to manually go back in and change it to 387 for it to calculate correctly.
Thanks!,
DS
Re: Efficient tuning of XGBoost parametersHi @DS ,
Thanks for continuing to pursue this. I just recently found and fixed a couple of problems with formula metadata that may currently be adversely affecting XGBoost results in the Model Comparison platform. These and several more updates will be available in JMP Pro 16 Early Adopter due out in a couple of months. In the meantime, you could try some basic comparisons with the Boosted Tree platform directly in your JMP table.
The following parameters map directly between Boosted Tree and XGBoost:
- Number of Layers <-> iterations
- Splits per Tree <-> max_depth
- Learning Rate <-> learning_rate
- Minimum Size Split <-> min_child_weight
- Row Sampling Rate <-> subsampling
- Column Sampling Rate <-> colsample_bytree
It may help a little further to change XGBoost advanced parameter grow_policy to "lossguide". As you indicate, try fitting and saving predictions from two similar models then plot the predictions against each other. They should be highly correlated.
The Tuning Design Table under Advanced Options is as you guessed. You need to set up a separate JMP table with column names exactly matching XGBoost parameter names then manually type in the name of this JMP table into the parameter box. You can create the table from scratch manually or right click in the Model Comparison table > Save as Data Table to get started. This enables specification of arbitrary designs across all parameters, but has only been lightly tested at this point so make sure to check results carefully.
Re: Efficient tuning of XGBoost parametersHi @russ_wolfinger,
Thanks for the information. I found this post to be quite helpful, at least in going through a direct example and iterative step-by-step approach to tuning XGBoost.
https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/
My particular situation has about 3500 training rows and 1000 test rows. As mentioned before, my test rows are in a separate data table that I concatenate to the first and then compare models -- this is to find out which model actually predicts best on an unused data set. Also, I split the 3500 training rows into 80/20 train/validate rows, stratified by my Y output I'm modeling.
I will work through some different tuning schemes to see what works best. But, one thing I'd like to do is verify that XGBoost can give me the "same" output as, e.g. bootstrap forest. Is it possible to take the parameters from either bootstrap forest or boosted tree and convert them to the appropriate settings in XGBoost? In principle, XGBoost should give nearly the same result, but much faster. I've tried to read some posts on how to do this, but I'm still struggling with how to translate the settings in the other platforms to XGBoost.
Is it possible that the compare model platform evaluates the XGBoost model differently?
Some more clarifying questions:
Is there a way to determine the number of trees from the XGBoost platform? Is this the field "iterations"?
Can we "feed" the platform our own tuning design table? Under the Advanced Options, there is a field with that name, but it's empty. If we have our own tuning table, can we enter the name of it there?
Thanks,
DS
Re: Efficient tuning of XGBoost parametersHi @DS , Thanks very much for the feedback. There are some decent manual methods for tuning. I just tried googling "tuning xgboost" and the first several links looks reasonable. These may be considered a bit more of an art than science, but they help illuminate what each parameter is doing and build an overall intuition about how XGBoost works.
Am curious to learn of your setup in case you get the chance to share or publish it. In the many comparisons I've done as well as in competitive situations like Kaggle I've found XGBoost to perform better. The performance gains tend to become stronger as the data set size increases.
Sounds like you are doing a classic training-validation-test split. It might be worth running this procedure several times with different splits to determine how stable your performance estimates are.
One advantage of k-fold is you get a "bagged" predictor over the k model fits, and it will almost always perform better than a single model trained on a portion of the data. An alternative approach is to do tuning with whatever validation approach you like then do one final fit at the end using the entire training+validation data.
