Understanding and Applying Tree-based Methods for Predictor Screening and Modeling

Question

JMP Mastering Tree based Models 17Aug2018 1_03_26 PM.zip

Georg · Answer

Thanks both for the very detailed analysis, I need some time to work on all your proposals, and will respond later. I know that there are many challenges in my dataset, one additional is that it is unbalanced, it is a mixture of much historical data and some experiments added to it. And we developers want to get the information out of all data, and not to throw anything. These experiments may look like outliers, but are not outliers, they are single points with a meaningful result. As my dataset will be presented in the discovery summit Europe, we can further discuss when it's online. BR

Victor_G · Answer

@Peter_Hersh : I am not sure what you imply with your sentence "Tree-based models are very susceptible to outliers", since tree-based method are instead quite robust to outliers, or at least outliers have a negligible effect. There is already a lot of examples and publications on this aspect, and this is also why this type of models are often used and famous, because of their "simplicity" and low pre-processing/cleaning step (no big work needed on outliers, and handling of missing values, no assumptions on distributions, etc...). A comparison of the impact of outliers on regression models and random forest (and the benefits of different techniques to deal with outliers applied on both algorithms (spoiler: which show no big effect on random forest algorithm)) can be seen here : How to Make Your Machine Learning Models Robust to Outliers - KDnuggets

@Peter_Hersh @Georg There are however other things to take into account for the choice of tree-based models, here are some :

* Overfitting that can occur mainly for decision tree and boosted tree (if no proper validation is done). Random Forests tend to be less prone to overfitting due to the bootstrapping process : there is an independent training of individual trees, each of which is fit to a bootstrap sample of the training data. With the bootstrapping process, some samples are not "used" in the creation of individual trees (called "Out Of Bag" (OOB) samples) but can be used to estimate errors in the model (and hence serve as "validation", estimation of error on data not seen by the model).

* Correlation/multicollinearity : The different tree-based models deal differently with multicollinearity or correlation between variables. If you have strong correlation/multicollinearity in your dataset, you may expect some differences : When looking at the Decision Tree features importance plot (or Gradient boosted Tree), you can see that only the "best" features are selected (the ones with the best split attributes), and the other ones (correlated to the first ones) are not selected and have feature importance of 0 (or close to 0). This is normal, since a Decision Tree use all the predictors and try to determine the best splits at each level, so "worse" and correlated features will be dropped out. So reloading Decision/Boosted Tree may lead to different models in presence of correlation/multicollinearity (as it may not always select the same features/variables each time). When looking at Random Forest features importance plot, you can see that all features have a score, even correlated ones. This is due to the feature bagging, a random selection of features done for each decision tree in the forest, giving each features the same "chance" to be selected in individual trees. So reloading this model should give you very similar models.

* Noise (or Bias/variance tradeoff): Boosted trees can be more accurate than random forests. In boosted tree, each individual tree is iteratively trained to correct previous tree's errors, so this algorithm is capable of capturing complex patterns in the data. However, if the data are noisy, the boosted trees may overfit and start modeling the noise. So depending on your dataset size and the noise in your data, there is a choice to make between Boosted Tree and Random Forest: The first ones are very accurate but not robust to noise (low bias, high variance like Decision Tree), whereas in comparison RF are less accurate/precise but are more robust to noise (bias is the same as each individual tree in the Forest, lower variance than Decision tree or Boosted Tree because of the bagging process).

And you also have to consider (in addition to previous points) your objective (predictive/explanative ?), the influence of the metrics used to evaluate the performances of your model on this dataset, the dataset size and dimensions (number of variables vs. number of observations), representativeness of your data, etc...

If you're using a small dataset (with possible noise), Random Forest may be a safer and more robust alternative than Boosted Tree/Decision Tree.

In addition to what @Peter_Hersh suggests, if you have different performances accross the folds/validation columns for your Boosted Tree, that is an indication that a subset of your data may have unusual values, so trying to look at distances (Jackknife/Mahalanobis) in the Multivariate platform could enable you to spot and identify an usual observation in this subset.

Sorry for the long answer, I hope it will be helpful,

Peter_Hersh · Answer

@Georg Great question.  @Victor_G has a couple of nice solutions with model averaging or ensemble modeling of k-folds.  It sounds like your data has a lot of noise with the model changing with additional runs.  It might be the case that the model is not very useful at this point without some additional factors or runs to help understand the variability in your data better.

A way to test this is to make multiple validation columns for the holdback validation.  If those models using different validation columns are dramatically different from each other that indicates that a random subset of your data is not representative of the data as a whole.  That means you will have trouble making an accurate predictive model without making some changes.

Tree-based models are very susceptible to outliers, so it might be a good idea to check for outliers before running the model.

Hope this is helpful let us know how things go.