nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Using Stratification and Grouping in Validation Tool

MikeD_Anderson

I've got an interesting question from a marketing company. They have a pool of stores - to make the math easier let's say 5000. Of the 5000 stores, 2000 are designated as "control" stores - they are off limits to advertising campaigns, etc. The other 3000 are "test" stores. The problem they are facing is that they want to select 200 stores from the control and test groups with similar demographics.

I think I found a way to do this by using stratification columns and grouping columns in the Validation Column Tool, and pairing that with the random stratified subsample capabilities in Subset. But, that seems a little complex and I'm wondering if someone has run into this before. It also has the problem that using stratification and grouping columns in the Validation Column Tool only allows you to have one column in stratify and one group column - they have between 3 and 5 that they want to consider for stratification.

Thoughts?

Best,

Accepted answers

All comments

DonMcCormack

Have you considered using one of the clustering tools and pairing/matching stores from the same group?

MikeD_Anderson

Yeah - I had them try hierarchical first to see if they could use the clusters in place of all the demographic information, but the distributions from some of the categorical variables (like the store brand distribution) weren't as close to matching as they'd like. It might just be that the population they're sampling from doesn't have enough stores to make two identical subsets from the test and control pools. ¯\_(ツ)_/¯

jiancao

In addition to the clustering scoring for the selection of a comparison group that Don mentioned, Another popular method is the propensity score matching https://en.wikipedia.org/wiki/Propensity_score_matching and https://www.scalestatistics.com/propensity-score-matching.html

SAS has a Proc dedicated to this method this https://go.documentation.sas.com/doc/en/statug/15.2/statug_psmatch_overview.htm

DonMcCormack

So you're matching two like groups and not store to store?

MikeD_Anderson

Correct.

Bill_Worley

@MikeD_Anderson have you thought of or tried using covariate DOE? You select the factors to be part of the "DOE" and let the DOE platform select the rows to be part of the subset. You may have to subset the Test and Control stores into separate tables to get this technique to pull 200 from each group, but I can imagine, with a little trial and error of course , that there would be a way to make this happen in one table.

Best,

Bill

ChristianStopp

Thanks @Bill_Worley - I'd had a question from a user about tools in JMP to create balanced groups, e.g. initial weight of mice assigned to cages in groups of 5, and had inferred this was one application of the covariate DoE with cage as block and initial weight as covariate. A little scripting to test out the hypothesis was initial encouragement I was on the right track.

(Admittedly I don't recall seeing the word 'balanced' in the reading I dug up in our docs so was hesitant.)

jiancao

Bill, thanks for bringing the covariates in Customer DoE to my (probably many others’) attention. This is a viable approach to generate both test group and control group where the distributions of covariates would be similar across the two groups. Below is what I got with a 50:50 random split of 500 subjects after using age, height, weight as covariates:

In comparison, the propensity score matching method I mentioned above is used in an observational study where treatment has been already applied to subjects but no control group is established. PSM can be used to find a group of untreated individuals from the population that’re comparable on observed covariates to the treated, and the treatment effect can then be estimated.