using Validation data on boosted treeI developed a predictive model w boosted tree. I had 30,000 rows of training data. I have 10,000 tows of “validation” data. How do i run the validation data set through the model ( I have over 800 variables in my model, so getting a “formula” was not practical ). Please provide an interactive solution (not script).Design of ExperimentsRe: using Validation data on boosted treeMy preference would be to append the test data (using Concatenate - after saving the prediction formula column and checking the box to evaluate formulas). You don't want to make the test data a validation set because it will be used in developing the models. JMP will automatically separate the performance of your model between the training, validation, and test data.
I am still concerned about your repeated references to the complexity of your model. Normally, JMP can handle quite complex formula columns. If it is complex enough to be causing issues with JMP evaluating it, then I worry about whether you should be using that model. If you had millions of rows, it could be a hardware limitation - but with 30,000 rows, there shouldn't be a problem unless you have too many variables in your model. As you suggest, you might have something like 800 variables in your model and I suspect this might cause issues for some predictive models. However, my initial reaction would be that you really don't want a model with 800 factors and only 30,000 data points (less if you are using validation and test data sets). Surely, you can get a model with almost the same performance using fewer factors. So, I think the first step would be to get a more manageable model.
Re: using Validation data on boosted treeHere's my thinking at this stage. If you've got a model in a JMP data table that you want to evaluate, then just append the new set of 'test' predictors and the response to the data table containing the model. I'd give the new set of rows a unique plotting symbol so in any plots you generate you'll know those are the 'new' rows. The 'predicted' column will populate with values for the 'test' predictors and then I'd just use Graph Builder for actual vs. predicted values and see what you see. Maybe put a local data filter in place for rows used in 'training' and 'test'?
Re: using Validation data on boosted treeThis sounds a little confusing, but I will try next week when back online.
thank you.
Re: using Validation data on boosted treeAha !
Re: using Validation data on boosted treeQuick Q -I have a Completely Separate Test data set. Do I combine it with
the original data set (i.e., append it) - to then run it through my model ?
If I do that - do I hardcode /create a validation column with 1s for Test
and 0s for training (or validation) to make the model run it ? I'm not too
keen on creating the "formula" - because it seems to be too big for the
program to manage. Does this thinking make sense ?
I really just want to be able to run the TEST set from the model and don't
quite know how. (Also, I will be working on it Next Week (not this week,
unfortunately) - since I have visiting family.
My thanks
Re: using Validation data on boosted treeOne small further clarification to add to @dale_lehman 's second reply. He is correct in stating that when you create a column used for row validation category specification, ie., a Validation Column, then all will be handled automatically by the analysis platform when you run the platform. Results will be included in the various platform report elements for the training, validation, and test subsets of data. The part Dale left out was when you create that Validation column you have to articulate in the dialogue box that you want to create all three categories, training, validation and test as well as the method by which each row is assigned. Sounds like you have a pretty large number of observations so a purely random assignment is probably a good place to start.
Re: using Validation data on boosted treeYour reply doesnt help, but thank you.
Re: using Validation data on boosted treePredictor Screening is a good first step, to understand, which variables are important. See script from Scripting index, to understand how it works.
I first would do a sampling of the original table to reduce number of rows (main menu: Tables->subset …)
With this subset you can try several Things in short time, save the scripts for the promising ideas and apply them to the large dataset later.
Names Default To Here( 1 );
dt = Open( "$SAMPLE_DATA/Boston Housing.jmp" );
obj = dt << Predictor Screening( Y( :chas ), X( :crim, :indus, :nox, :rooms, :age, :distance, :radial ) );
Re: using Validation data on boosted treeIf you create the validation column using the facility in the Predictive Modeling Platform, then JMP automatically runs your model on the training, validation, and test data. So, you don't need to do anything further. Now, your comments about problems getting your model to run are a different matter - it sounds like there are issues with the model,and I suspect that having 800 variables in your model reflects something that needs to be corrected. I would look for ways to reduce the number of features.
Re: using Validation data on boosted treeI will try that.
However, because the model is so large the prediction formula didnt seem like it would even run at all- it seems to stall the program. Lastly, when i do reallt want to run the test data- how do i run it ?? Thank you so much - ksfire1
Re: using Validation data on boosted treeI don't understand your question. If you have a validation column and put that in the Validation box, then the model results for the validation data are reported. I do second the opinion that you might want a simpler model and that you should try different models. I would add that you might want to set aside 10% of the data as test data, since the validation data is used in building the model and there is a danger of overfitting, particularly with 800 variables in your model. So, I would suggest breaking your data into training, validation, and test data sets, running a variety of models, and using the Model Comparison feature to see how the models perform on the test data. This comparison should be automated once you have saved the prediction formulas from all of your models.
Re: using Validation data on boosted treeWith the formula all would be easy. Did you try? What happend?
I assume that you use JMP Pro.
Did you try to build a less complex model? Did you try Predictor Screening or other variable reduction techniques? Or other modeling techniques?
To have the best model, one should have compared different techniques/variable sets.
