how to know what rows are my validation dataset when I run model with random split (JMP PRO)

Question

How can I identify which rows(observations) are my validation points when my target variable is category variable when I use random selections? such as example from https://www.jmp.com/support/help/14/example-of-profit-matrix-and-decision-matrix-rep.shtml#1276835 I know for continuous target variable, I could select rows from predict vs actual plot to know which observations are labeled as validation/training data. but I could not find this for categorical target variable. thanks

Peter_Bartell · Accepted Answer

I think I see what you are doing...but I'm not 100% sure. It sounds like you might be selecting the validation % from the modeling platform dialogue launch? What I am suggesting is BEFORE modeling, using JMP Pro's Make Validation Column capability, create a column in the data table whose row values will be training, validation, and optionally, test. Then within the model launch window just place the validation column name in the Validation role and then run the platform as you would normally. See the attached example data table for what I'm suggesting...I didn't include predictor variable columns but it shows what I was suggesting in my first reply.

Jeff_Perkinson · Accepted Answer

You're using the "Validation Portion" option in the Partition launch dialog.

Unfortunately, I don't see a way to identifiy which rows are used for validation with this method.

If you've got JMP Pro, I agree with @Peter_Bartell. You can create a Validation column to specify which rows are used for validation.

Peter_Bartell · Accepted Answer

The other thing I'll suggest is if you have a categorical response, a general best practice wrt to creating the Validation column in JMP Pro's Make Validation Column capability is to select the Stratified Random option circled below, then select your target variable. This will force the randomization % you choose to apply to each level of the categorical variable...which helps prevent against model bias.