Challenging Bootstrap Forest in JMPHi,
a customer of mine is challengin Bootstrap Forest in JMP being too inflexible of changing settings fo rthe algorithms. He compares it with H2O (http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/drf.html) . In the attached document I've tried to match JMP's settings with their's (in grey, the yellow ones are only in H2O).
There are several settings not possible to change (and probably not implemented) in JMP, like e.g.
- offset_column (Offsets are per-row “bias values” that are used during model training)
- balance_classes (Oversample the minority classes to balance the class distribution)
- nbins (Specify the number of bins for the histogram to build, then split at the best point)
- stopping_metric (Many more than R2 and RMSE: Specify the metric to use for early stopping
- checkpoint (Use this option to build a new model as a continuation of a previously generated model)
- col_sample_rate_per_tree/level (Specify the column sample rate per tree or level)
- min_split_improvement (The value of this option specifies the minimum relative improvement in squared error reduction in order for a split to happen -> can help reduce overfitting)
- and some more
I've asked him what he expects from having these options as I doubt the resulting model will bring that much improvement that it counts as serious. He states it is remarkable better. I then checked the H2O site and guess it is due to a very special problem and for that you might really get a better result using all those tuning options.
Do you have heard from other customers having said that there are other (better tools) for random forest models out there?
And, Chris or other developers: do you think adding the one or other setting would be a good option?
Thanks,
Martin
Re: Challenging Bootstrap Forest in JMPMartin,
This H2O pacakge is new to me, but it looks like I can use it with python and in an ipython notebook too.
This link on git hub makes it look like it might be relatively easy to use once its setup too.
https://github.com/h2oai/h2o-3/blob/master/h2o-docs/src/product/tutorials/rf/rf.md If I have time, I'd like to see if I can run with it. It might be a little easier to use than XGBoost. Both of them have some grid search options for tuning.
"H2O is an open source, in-memory, distributed, fast, and scalable machine learning and predictive analytics platform that allows you to build machine learning models on big data and provides easy productionalization of those models in an enterprise environment." How its described on the main web page, is remarkablly similar to the language used to describe SAS Viya. If your customer wants this, maybe they've outgrown JMP?
Re: Challenging Bootstrap Forest in JMPThe ability to change stopping metric is something that has been requested by a couple customers in my patch. Balance classes can be achieved be inverse proportion weighting, but would be nice if it was automated. Thanks for the heads up on this.Re: Challenging Bootstrap Forest in JMPput it into the jmpbeta@jmp.com email address for the development team to have a look. I'd also point out that this might be a specialized subset of Random Forest. They could just be tacking that "Distributed" bit to avoid the copyright issue, but it could also be indicative of something different. I've run into another one called "Conditional Tree" that is a specialized flavor of Gradient Boosted Tree...
FWIW, Most of the data science guys I talk to are content with what we have. Some of the options that they mention above sound like they could be handled with weighting and different validation strategies.
MRe: Challenging Bootstrap Forest in JMPOne more comment on this: I ave heard that more tuning options thing more often for Neural Networks, like more layers, more training and activation algorithms to be set, ...
But it was new to me about Random Forest. So I was also curious if others have heard similar stories.
