nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Random subset selection for a given distribution

altug_bayram

Hello ...

I am trying to randomly select from a column of data for a given normal distribution . The given distribution is different than that of the data.

Is there an easy way of accomplishing this with Table Subset's current options ? I could not see anything that could help me.

I also tried a formula, basically NormalDensity (X, mu, sigma) to plot the densities of the "data" against the given distribution.

But I still don't know how to say uniformly & randomly select from this formula so my outcome is as close as possible to the given distribution.

Thx. Appreciate the support.

Accepted answers

ian_jmp

If "So bottom line, I need to sub-select from 400K+ rows a number of sub-samples so that the distribution for the selected sub-sample is as close as possible to the given distribution" is what you want, maybe you could use this kind of approach, or something close to it?

NamesDefaultToHere(1);

// Make a big table
dt1 = NewTable("Big", NewColumn("Values", Numeric, Continuous, Formula(RandomUniform(-3,3))));
dt1 << addRows(40000);
dt1 << runFormulas;
Column(dt1, "Values") << deleteFormula;

// Make a small table
dt2 = NewTable("Small", NewColumn("Values", Numeric, Continuous, Formula(RandomNormal(0,1))));
dt2 << addRows(40);
dt2 << runFormulas;
Column(dt2, "Values") << deleteFormula;

// For each value in dt2, find a value in dt1 that's closest
bv = Column(dt1, "Values") << getValues;
sv = Column(dt2, "Values") << getValues;
selectedRows = J(NRow(sv), 1, .);
for(i=1, i<=NRow(sv), i++,
	dist = (bv - sv[i])^2;
	selectedRows[i] = LocMin(dist);
);

// Subset dt1
dt3 = dt1 << subset(rows(selectedRows), LinkToOriginalDataTable(1));

// Look at the distributions
NewWindow("Compare Distributions",
	HListBox(
		dt1 << Distribution( Continuous Distribution( Column( :Values ) ) ),
		dt2 << Distribution( Continuous Distribution( Column( :Values ) ) ),
		dt3 << Distribution( Continuous Distribution( Column( :Values ) ) )
		)
	);

Jeff_Perkinson

The approach that @ian_jmp found was the first thing I tried, but I abandoned it too early apparently as Ian found a way to overcome some shortcomings that I ran into. Nice job.

As for eliminating resampling of rows, I believe that Ian was suggesting adding the line:

	bv[Loc Min( dist )] = .;

to the for() loop. Thusly:

Names Default To Here( 1 );

// Make a big table
dt1 = New Table( "Big",
	New Column( "Values", Numeric, Continuous, Formula( Random Uniform( -3, 3 ) ) ),
	New Column("Row", Formula(Row()))
);
dt1 << addRows( 40000 );
dt1 << runFormulas;
Column( dt1, "Values" ) << deleteFormula;

// Make a small table
dt2 = New Table( "Small",
	New Column( "Values", Numeric, Continuous, Formula( Random Normal( 0, 1 ) ) )
);
dt2 << addRows( 40 );
dt2 << runFormulas;
Column( dt2, "Values" ) << deleteFormula;

// For each value in dt2, find a value in dt1 that's closest
bv = Column( dt1, "Values" ) << getValues;
sv = Column( dt2, "Values" ) << getValues;
selectedRows = J( N Row( sv ), 1, . );
For( i = 1, i <= N Row( sv ), i++,
	dist = (bv - sv[i]) ^ 2;
	selectedRows[i] = Loc Min( dist );
	//the line below prevents rows from the "Big" table from being re-sampled
	bv[Loc Min( dist )] = .;
);

// Subset dt1
dt3 = dt1 << subset( rows( selectedRows ), LinkToOriginalDataTable( 1 ) );

// Look at the distributions
New Window( "Compare Distributions",
	H List Box(
		dt1 << Distribution( Continuous Distribution( Column( :Values ) ) ),
		dt2 << Distribution( Continuous Distribution( Column( :Values ) ) ),
		dt3 << Distribution( Continuous Distribution( Column( :Values ) ) )
	)
);

All comments

Peter_Bartell

Once you have your population column from which you want to sample, try out, this workflow:

1. Select the column from which you want to sample.

2. From the JMP main menu bar, select Rows -> Row Selection -> Select Randomly -> and then your sample size or sampling rate.

This will return selected columns in your original data table, and from there Row Subset will give you a data table.

altug_bayram

Peter .. thanks for your suggestion.
but that does not seem to work.

My original data's mean/stdev are: 12816 and 1726 from a very large sample (400K +).
I am trying to downselect a sub-sample that is closer to 14300 and 2050

I tried your suggestion on the formula (i.e. densities of the data wrt 14300, 2050) and unless I can force this random dist to be uniformly selected among the densities, I don't think I can get what I need. In fact, sub-sampling came out as expected w/ the original distribution.

Peter_Bartell

I see...I did not fully understand your original post. Your right my suggestion will just return a distribution with very similar parameters to the original distribution. If you want observations of a random variable with the desired parameter estimates, why can't you just create a column with those parameters and move on. Sure, the specific observations won't necessarily reside in the original column, but that may or may not influence what you're trying to acheive analytically or practically? They'll at least be plausible observations, though not exact matches.

altug_bayram

My real goal is to compare N (other) parameters from two data sets already available, I am not trying to create a fictitious distribution. (Maybe I am misunderstanding your proposal).

The given distribution is a constraint on an important parameter that may influences others. I am trying to select assets w/ similar distribution from "data" to a given distribution so to minimize the impact of this influential parameter. In this context, "distribution" only applies for a single parameter, and not multiple.

So bottom line, I need to sub-select from 400K+ rows a number of sub-samples so that the distribution for the selected sub-sample is as close as possible to the given distribution.