246045_en-US

liliconcom

Simulate functionality for XGBoost

Hi All,

 

  I would like to take advantage of the "Simulate" functionality for JMP statistics outputs within XGBoost (value of interest > right click > simulate). Unfortunately, whenever I do, I get a JMP Alert that says: "Unable to perform simulation. No switchable columns." However, that is not correct. I have function columns I'd like to switch out for simulating a value, e.g. subsample or iterations, but I can't.

 

  Is this function disabled for XGBoost? Or is it part of the known problem #4: JSL scripting is not fully functional?

 

  Just want to understand where that error is coming from since I can run simulate functionality on other tree platforms within JMP.

 

Thanks,
DS

Re: Simulate functionality for XGBoost

Thanks Mike, as time permits we should try some autovalidation with XGBoost.

Re: Simulate functionality for XGBoost

Ah yes, apologies @DS, right-clicking on a table column > Simulate  or Bootstrap is not currently working but will be coming in JMP 16 Pro Early Adopter due out in another month or two.   In the mean time your only option is manual looping and extraction with JSL, which I think you have going.    Glad you are making investigations along these lines.

Re: Simulate functionality for XGBoost

Hi @russ_wolfinger, 

 

  Thanks for the feedback, but I think what you are referring to is different from what I was trying to describe. After performing a fit, see for example the images attached (I've deleted the actual column names and so forth from the image), one can right click on a table output, and select Simulate. From there, you get a choice to swap out a given column with another. For example, I can have a new validation column that randomly selects 80% for training and the additional 20% for validation. 

 

  As an example, after using a small tuning table for a bootstrap forest fit, I then did a small simulation on the training and validation R^2 by swapping out the original validation column (which was originally stratified by the response, Y) with the random generated 80/20 column. By doing this, it should be able to test the model against randomly chosen validation sets that are not constrained by stratification, thereby testing how robust the model is across randomly chosen train and validation sets. It's faster than swapping out new validation columns.

 

  My Y-variable is mostly normally distributed. Interesting idea to generate a random normal distribution from the model fit. That is not too different from the Simulator functionality under the Profiler, but a much faster way than generating it. Unfortunately, it's not really what I'm after.

 

  This Simulate feature seems to have a bug in the XGBoost add-in where it doesn't recognize any switchable columns. It doesn't pull up the Simulation window to swap out columns.

Thanks, DS

Re: Simulate functionality for XGBoostHey, All!

Something I've done the past with Tree models is to use a random number generator in the weight column using Chris and Phil R.'s Autovalidation workflow (https://community.jmp.com/t5/JMP-Add-Ins/Add-in-To-Support-Auto-Validation-Workflow/ta-p/189991).

Best,

MRe: Simulate functionality for XGBoost

Hi @DS ,

 

Simulate functionality requires the formula to contain some kind of random number generator.   As far as I can tell, none of the tree-based methods in JMP Pro provide formulas such as this, but you can get them in places like Generalized Regression with Save Columns > Save Simulation Formula. 

 

A relevant research question is:  What is a sensible data generating mechanism behind a boosted tree model?   Assuming your Y variable is approximately normally distributed,  one easy thing to try would be to wrap the XGBoost prediction formula with a Random Normal() call, using the prediction as the mean part.   You would also need to specify something reasonable for the standard deviation, an obvious choice being the out-of-fold root mean squared error.   Similarly, if Y is binary, put the XGBoost formula inside of Random Binomial() as p and specify n = 1.

 

You may be one of the first people to conduct simulations along these lines and the results are likely publishable.