Hi,
I'm trying to extract some of the statistics from the XGBoost report using JSL code (rather than right click > make into data table). I do this with some of the other platforms (e.g. boosted tree and bootstrap forest) without any issues, but for some reason I'm having a hard time getting it to work in XGBoost.
In the other platforms, I basically have a script in the table I run that uses a tuning table, but fits them one at a time and extracts the statistics as it goes along. But modifying a similar approach to work with XGBoost is not working. Is there something in the XGBoost report that would preventing this?
As an example, for bootstrap forest modeling, I can do something like the following:
Names Default To Here( 1 );
dt = Data Table( " " ); //this would be your data table used to train the model
dt_pars = Data Table( " " ); //this would be your tuning table with the parameters in it.
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) ); //this basically duplicates the previous table for writing stats values
dt_results << New Column("Stat 1"); //for recording whichever stat you're interested in, e.g. Training R^2
dt_results << New Column("Stat 2"); //for recording whichever stat you're interested in, e.g. Validation R^2
//and so on with however many stats you want to get.
//This For Loop will run each row of the tuning table iteratively. This is really helpful for platforms that use a lot of RAM.
For( i = 1, i <= N Row( dt_pars ), i++,
NumTrees = dt_pars:Number Trees[i];
NumTerms = dt_pars:Number Terms[i];
PortBoot = dt_pars:Portion Bootstrap[i];
MinSplpTr = dt_pars:Minimum Splits per Tree[i];
MaxSplpTr = dt_pars:Maximum Splits per Tree[i];
MinSplSz = dt_pars:Minimum Size Split[i];
str = Eval Insert(
"report = (dt << Bootstrap Forest(
Y( :Y1 ), //whatever your response column is
X( :X1, :X2, :X3), //this is the list of all the x-factors
Minimum Size Split( ^MinSplSz^ ),
Validation Portion( 0.2 ),
Set Random Seed( ),
Method( \!"Bootstrap Forest\!" ),
Minimum Splits per Tree( ^MinSplpTr^ ),
Maximum Splits per Tree( ^MaxSplpTr^ ),
Portion Bootstrap( ^PortBoot^ ),
Number Terms( ^NumTerms^ ),
Number Trees( ^NumTrees^ ),
Early Stopping( 0 ),
Go,
invisible
)) << Report;"
);
Eval( Parse( str ) );
Stat_1 = report["Overall Statistics"][Number Col Box( 1 )] << Get; <U><STRONG>//this is where it doesn't work for the XGBoost report</STRONG></U>
//and so on for any other stats of interest
report << Close Window;
dt_results:Name("Stat 1")[i]=Stat_1[1];//saves the first in the list of Stat_1
dt_results:Name("Stat 2")[i]=Stat_1[2];//saves the second in the list of Stat_1
//continue on until you've recorded all the stats of interest in the dt_results data table.
);The above method works really well if you want to run MANY bootstrap forest runs while not consuming too much RAM. You can easily record the RSquare, RASE, N, and so forth from the "Overall Statistics" outline box.
When I run a similar script (below), it throws an error at the: report["Outline Box"][Number Col Box(1)] << Get line.
Names Default To Here( 1 );
dt = Data Table( " " ); //training data table
dt_pars = Data Table( " " ); //tuning parameter table
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) ); //makes duplicate for writing output values to
//creates columns for the statistics to save in the results data table.
dt_results << New Column( "Stat1" );
dt_results << New Column( "Stat2" );
dt_results << New Column( "Stat3" );
dt_results << New Column( "Stat4" );
dt_results << New Column( "Stat5" );
dt_results << New Column( "Stat6 );
//this is the For Loop to run each row in the dt_pars tuning table one at a time.
For( i = 1, i <= N Row( dt_pars ), i++,
maxd = dt_pars:max_depth[i];
subsam = dt_pars:subsample[i];
colsambytree = dt_pars:colsample_bytree[i];
minchweight = dt_pars:min_child_weight[i];
al = dt_pars:alpha[i];
lam = dt_pars:lambda[i];
learnr = dt_pars:learning_rate[i];
iter = dt_pars:iterations[i];
str = Eval Insert(
"report = (dt << XGBoost(
Y( :Y1 ), //Response column
X( :X1, :X2, :X3, :X4),
Validation( :Validation), //validation column of choice
Fit(
max_depth( ^maxd^ ),
subsample( ^subsam^ ),
colsample_bytree( ^colsambytree^ ),
min_child_weight( ^minchweight^ ),
alpha( ^al^ ),
lambda( ^lam^ ),
learning_rate( ^learnr^ ),
iterations( ^iter^ )
),
SendToReport(
Dispatch( {}, \!"Launch\!", OutlineBox, {Close( 1 )} ),
Dispatch(
{\!"Launch\!", \!"Advanced Options\!"},
\!"\!",
TextEditBox,
{Fixed Size( 1, 72, 19 )}
),
Dispatch(
{\!"Launch\!", \!"Advanced Options\!"},
\!"\!",
Text Edit Box( 2 ),
{Fixed Size( 1, 72, 19 )}
),
Dispatch(
{\!"Launch\!", \!"Advanced Options\!"},
\!"\!",
Text Edit Box( 3 ),
{Fixed Size( 1, 72, 19 )}
),
Dispatch( {\!"Launch\!"}, \!"\!", PictureBox, {Set Summary Behavior( \!"Collapse\!" )} ),
Dispatch( {}, \!"Model 1\!", OutlineBox, {Close( 1 )} ),
Go,
invisible
)) << Report;"
);
Eval( Parse( str ) );
Stat_1= report["Compare"][Number Col Box(1)] << Get;
//and so on for the other stats to be saved
report << Close Window;
dt_results:Name( "Stat1" )[i] = Stat_1;//saves the value to the data table dt_results
//and so on for the other stats to record in the results data table.
); I'm trying to get the stats for each fit that come up in the "Compare" [Table Box(1) ], like Train RSquare and so forth, but the reference to the Number Col Box seems to not be working -- it doesn't matter what value is in () 1, 2, 3, 4, etc. they all throw the same error. I'd like to save the RSqaure, Corr, and RMSE values for training and validation after each fit. Even if I try it under the "Model 1" outline box, child "Actual by Predicted", and look for the Number Col Box() -- e.g. (12 or 13) -- for the training results and validation results, and try to extract the values there, I still get an error that is thrown in the log file.
The error I get is:
object not subscriptable in access or evaluation of 'report["Compare"][Number Col Box(1)]' , report["Compare"][/*###*/Number Col Box( 1 )]
In the following script, error marked by /*###*/
Stat_1= report["Compare"][/*###*/Number Col Box( 1 )] << Get
I know I'm not missing any parentheses or "glue" ; in the script.
I don't understand why I can extract values from other reports, but can't seem to do it with the XGBoost report. Is there something I'm missing in the JSL code?
I'm thinking getting the numbers from the "Model 1" outline box will be better because the "Compare" outline box doesn't always show the Corr and RMSE values -- unless you modify that with the hot button..
Any thoughts/suggestions on this are much appreciated.
Thanks in advance,
DS
