Tuning Table for estimating Partion Models: Results

MAS

An option for partitioning models (tree, bootstrap, boosted) is to use a tuning design table. The results begin with a table that includes the design points as well as the validation R-squared and the validation RMSE for each design point. I need to have that table also show the training R-squared, training RMSE, test R-squared, and test RMSE. Anyone able to help out here--maybe with a script?

 

These statistics, and more, are shown below the table for the chosen model, but I need these statistics for all design points. The method to my madness is this: I want to be able to produce a graph showing, say, both validation R2 and training R2 against learning rate to show the amount of overfitting that occurs with increasing learning rate.

cwillden

I change my mind.  It's not too difficult to script up.  Here's an example with boosted trees and the Car Physical Data from the sample data library.  I created a tuning table for this to try out, which I am attaching.  I also put a validation column in the Car Physical Data table.

Here's the script:

dt = Data Table("Car Physical Data");
dt_pars = Data Table("TuningTable");

//Get copy of pars table to store results
dt_results = dt_pars << Subset( All rows, Selected columns only( 0 ) );
dt_results << New Column("Rsq Training");
dt_results << New Column("Rsq Validation");

for(i = 1, i<= N Row(dt_pars), i++,
	SPT = dt_pars:SplitsPerTree[i];
	LR = dt_pars:Learning Rate[i];
	RSR = dt_pars:Row Sampling Rate[i];
	CSR = dt_pars:Column Sampling Rate[i];
	str = Eval Insert("\[report = (dt << Boosted Tree(
		Y( :Weight ),
		X(
			:Country,
			:Type,
			:Turning Circle,
			:Displacement,
			:Horsepower,
			:Gas Tank Size
		),
		Validation( :Validation ),
		Method( "Boosted Tree" ),
		Splits per Tree( ^SPT^ ),
		Number of Layers( 50 ),
		Learning Rate( ^LR^ ),
		Row Sampling Rate( ^RSR^ ),
		Column Sampling Rate( ^CSR^ ),
		Go,
		invisible
	)) << Report;]\");
	
	//Fit the boosted tree
	Eval(Parse(str));
	
	//Get the Rsq values and insert them into the results table
	R2 = report["Overall Statistics"][NumberColBox(1)] << Get;
	report << Close Window;
	dt_results:Rsq Training[i] = R2[1];
	dt_results:Rsq Validation[i] = R2[2];
);

I did a stack of the R2 columns for the results table to make the following graph:

 This should be pretty easy to adapt to the other platforms you mentioned.

 

TuningTable.jmp