Re: Splitting data that is not intended to be used as Validation columnsI've recorded this internally and added the defect id above.
@reese_sj, can you please enter a new item for the second part of this request (i.e., the out of bag estimates for model validation)? These are two different requests.
Thanks!
Splitting data that is not intended to be used as Validation columnsValidation Column: Quite often we need to split data in a training and test set, but not into a validation set that is used for model selection. This does not work in JMP. E.g. for modeling methods where we can use cross validation, like NN or GLR, and want to make the model selection/validation based on the training data and using the test data really only for testing the model. Further when test data are excluded, boosted trees and random forest, are using the excluded data for validating the model (possibly for NN). This requires extra steps to delete the test data for modeling and add them later again for model comparison.
Statistics and AnalyticsRe: Splitting data that is not intended to be used as Validation columns - Status changed to: Not Planned For Now@reese_sj is the concern that the validation data is cheating? How do you propose the model decides when it has fit the data the best? We often see just validation data with no test set, but not the reverse. JMP doesn't have a way to fit a NN without some form of validation or holdback. If you let the algorithm fit the training data as best as it can you will most likely end up with a overfit model. I would create the model and publish it to formula depot then run it on the test data set if you want to see how well it fits.
