Adding penalties to number of splits in Partition platform to reduce number of leaves?I have recently been spending a lot of time in the Partition platform, and currently I'm looking for a way to automate the analyses.
My issue is this: generally I've been splitting the data set 50/50 training/validation, and hitting "Go", which maximizes the validation RSquare. But, this often leads to models which have a larger number of splits than are really credible, e.g. here where it creates 5 splits to get to Validation RSquare 0.288, when just using two splits would get an almost-equally good fit of 0.27.
As a result, I generally have to always manually prune the splits to get an acceptably-good Validation RSquare with an acceptably-low number of model terms.
Is there a way to ask JMP to apply a penalty, and create some sort of "regularized" Validation RSquare that penalizes the model selection criteria based off the number of splits? E.g., the "regularized" Validation RSquare could be [Validation RSquare] - 0.05*[N Splits]. Or maybe there's a better way to regularize (e.g. continue adding terms if they can model at least 5% of the residual variance, rather than 5% of the absolute variance.)
JSL solutions are welcome, as this would primarily be for automated analyses.
AnalyticsScriptingRe: Adding penalties to number of splits in Partition platform to reduce number of leaves?Hi @markschwab ,
Regarding your concern, is it that the number of splits is really credible, or that it's a reasonably low number of splits? I believe the number of credible splits is the rows in the data table minus 1, but not 100% on that.
Anyway, I'm including a snippet of code that you can use to do your automated partitioning. I did this with the Boston Housing.jmp file.
If you're familiar with JSL, you should be able to modify it for your needs. The code is fairly basic, it's assuming you have a numeric, continuous response column. It runs the partitioning for however many max partitions you define from the first window and then saves the partitioning statistics. You can easily create your own column formula to have a "penalized" validation RSquare for example or however you choose to penalize the partition. You can then select which partition is the best for your criteria and re-run it with the fixed number of splits you determine. I have some ideas of how to fancy-up the program, but that's for later.
On a side note, you might consider creating a validation column (if you don't have Pro, you can do this by searching it in the Scripting Index) and stratifying by your response column, this will give the training/validation sets a more equal representation when dealing with non 50/50 splits. This way, you can have more data to train your partition.
Hope this helps!,
DS
Names Default To Here( 1 );
lbWidth = 168;
//dt = Open( "$SAMPLE_DATA/Boston Housing.jmp" );
// Expression to store the current settings in global variables
recallRolesS1 = Expr(
::dtchosenRecall = dtchosen
);
// Expression to clear all current settings. KW: Clear
clearRoles = Expr(
Try(
colListY << RemoveAll;
ColListX << RemoveAll;
ColListW << RemoveAll;
ColListF << RemoveAll;
colListV << RemoveAll;
)
);
// Expression to store the current settings in global variables
recallRoles = Expr(
::ycolRecall = colListY << GetItems;
::xcolRecall = colListX << GetItems;
::WcolRecall = colListW << GetItems;
::FcolRecall = colListF << GetItems;
::vcolRecall = colListV << GetItems;
);
//Function to choose the data table to model. KW: choose_data
choose_data_table = Function( {},
list = {};
For( i = 1, i <= N Table(), i++,
list[i] = Data Table( i ) << get name
);
win = New Window( "Select a data table",
<<MODAL></MODAL>Re: Adding penalties to number of splits in Partition platform to reduce number of leaves?Hi @markschwab ,
You are using the code as intended, I had just made some guesses that weren't generic enough to work under any circumstance. I thought you had a validation column, and so scripted it to run with one. the updated script below works around that by letting you input the validation portion, just like in the partition platform window. But, don't use both a validation column and a validation portion -- the script hasn't been coded yet to account for that and is again, not fully generic enough for all possible user inputs.
As for the error that you were getting, it was again because I assumed that there was more than one X factor in the process and therefore, scripted the reference to the report window accordingly. That is now fixed so that it doesn't matter if there's only 1 X or multiple X's.
Again, this script is not completely generic enough to account for non-numeric/non-continuous response column, Y. If the response is categorical, this script is not designed to account for that. It can easily be modified to do so if need be.
Hope this helps,
DS
UPDATE: Ok, so I couldn't help myself to code more and I included some penalized RSquare columns and a training-validation RSquare column. You can change the formula and the coefficient of the penalized calculation by editing the code -- I set it to your concept of Rsquare-0.05*Splits. In addition to that, I also included some code to graphically display the results when done running the calculations.
UPDATE 2: I went back and made the code a little more fun -- Now you can see the table results alongside the graph results and by selecting the point on the graph you like best, you can then re-run the partition for that specific number of splits when you click "OK". As before, I haven't made it super-generic for any kind of user input, but tried to make it work for numeric/continuous responses. What you now have should be enough to get you going.
P.S. As you learn the code, use the Scripting Index in JMP's Help menu. It's very helpful in understanding some of the calls and how to use them when scripting.
Names Default To Here( 1 );
lbWidth = 168;
dt = Open( "$SAMPLE_DATA/Boston Housing.jmp" );
// Expression to store the current settings in global variables
recallRolesS1 = Expr(
::dtchosenRecall = dtchosen
);
// Expression to clear all current settings. KW: Clear
clearRoles = Expr(
Try(
colListY << RemoveAll;
ColListX << RemoveAll;
ColListW << RemoveAll;
ColListF << RemoveAll;
colListV << RemoveAll;
)
);
// Expression to store the current settings in global variables
recallRoles = Expr(
::ycolRecall = colListY << GetItems;
::xcolRecall = colListX << GetItems;
::WcolRecall = colListW << GetItems;
::FcolRecall = colListF << GetItems;
::vcolRecall = colListV << GetItems;
::max_spRecall = max_sp_input << get;
::partVPRecall = partVP_input << Get;
);
//Function to choose the data table to model. KW: choose_data
choose_data_table = Function( {},
list = {};
For( i = 1, i <= N Table(), i++,
list[i] = Data Table( i ) << get name
);
win = New Window( "Select a data table",
<<MODAL></MODAL>
