nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Assign category to row keeping total probability fixed

SDF1

Dear JMP Community,

(WE10, 64-bit, JMP Pro 4.1.0)

I am trying to generate a column formula that performs an action similar to "create validation column" by assigning either 0, 1, or 2 to the row. I am trying the Random Category function as a way to do this, and it works, but not the way I want. It assigns either 0, 1, or 2 to the rows, but when you add up the frequency of each category, you get a value larger than 1, which I don't want.

In "create validation column", you can assign a certain percentage of your data table as "train", "validate", or "test" (0, 1, 2), and the sum of those fractions always adds to 1. I want to do this with a function, such as Random Category, so that the fraction of "0's", "1's", and "2's" add up to 1. For simplicity sake, I want to keep the number of "1's" equal to the number of "2's", i.e. the number of validate and test data are equal and add up to 1 - number of training data rows.

The purpose behind all of this is that I'd like to test out tuning a model by changing the number of train, validate, and test data sets in order to minimize the RMSE. This is done by right clicking the statistic of interest and then selecting "Simulate", which does bootstrapping and re-runs the model to evaluate how the statistic changes. BUT, you have to have a function to swap out for the number of rows in each of the validation data sets.

Each time it runs a simulation, it would "generate" a new validation column, but with a different ratio of "train", "validate", "test" rows. The output could then be used to find the optimal ratio of the three for having a better performing model.

Any thoughts/feedback on this is greatly appreciated.

Thanks!,

Accepted answers

Mark_Bailey

I would try this formula.

p = Random Uniform();

If( p <= 0.5, 0, p <= 0.8, 1, 2 );

I used 50% training, 30% validation, and 20% testing.

All comments

Mark_Bailey

I would try this formula.

p = Random Uniform();

If( p <= 0.5, 0, p <= 0.8, 1, 2 );

I used 50% training, 30% validation, and 20% testing.

SDF1

Hi @Mark_Bailey,

Thanks for the suggestion. This works, however I did modify it slightly as I don't want to have a fixed ratio of train, validate and test, but rather modify this each bootstrap simulate to see if there is an optimum ratio of train/validate/test, since the selections in a validation column can have an impact on the model fit, maybe a simple 0.6, 0.2, 0.2 split isn't optimum. Maybe it should be 0.72, 0.15, 0.13 for example.

p = Random Uniform();

If( p <= Random Uniform(x1,x2), 0,<BR />    p <= Random Uniform(X3,x4), 1,<BR />    p <= Random Uniform(x5,x6), 2 );

Adjusting the x1, x2, x3, x4, x5, and x6 can lead to a range of different ratios of the train/validate/test in the formula column.

Thanks!,

Mark_Bailey

You should only use one random variable at a time. There is no need to call Random Uniform() more than once, before calling the If() function. Use your variables for the Boolean expressions.

p = Random Uniform();

If( p <= x1, 0, p <= x2, 1, 2 );

SDF1

Hi @Mark_Bailey,

If I use the modified version, I get a wider range of ratios than if I use the condensed form.

The other way, I can see a wider range for the test training set (about 65%-78%) with the others distributed accordingly -- roughly 50/50 validation/test. This is really what I'm after: change the percentage of the test training set between a low value and upper value and then split the remainder 50/50: validation/test.

Thanks!,