Looking for a sample size calculator for defect proportions that can help me optimize the sample size with a minimum proportion to detect and the CIs for computed proportions.

students_t

Use case:

 

35,000 defects detected by a machine on one wafer. An operator samples 1500 images and sorts them into 10 categories. The percentages of the defect categories are meant to reflect the defects in the population of 35,000. In this case 1500 images were used but what happens when the 5000 images are used? 

What does that do to the confidence intervals on the proportions. As the sampling gets low, some of the low proportions will not be detectible. How does one determine the lowest detectible proportion for a given samples size.

 

Any ideas if there any calculators or tools that can help with this?

ih

You might be able to replicate this experiment within JMP.  If so, and if you have JMP Pro, you could do a Monte Carlo simulation to evaluate different sample sizes and the rate of detection for each.  This would be an alternative to more pure statistical methods which I'm sure exist.

 

Consider a 35,000 row table (one row per defect) with:

• A column of random data representing the actual category for each defect, weighted with some categories more important than others to represent your actual data.  This part requires a lot of attention and thought, you need this to represent the distribution(s) your categories might take. It might need to randomly choose from a variety of distributions.
• A sample size column, where the first row indicates how many rows to sample
• A column which 'selects' the prescribed number of rows
• A column with the 'detected' categories for each of the sampled rows
• A column to check whether the most common actual category matches the most common detected category (only use the first row of the column)

Use the distribution platform and include both the first column (actual defect category) and the last column (whether the correct defect category was identified).  Under the frequencies for the correct category column, you should see a single value.  Now for the JMP Pro magic: right click on a value in the table and select simulate. Swap out the Defect Actual Category column for itself. When it runs a new table will show up where each row represents a time your original table was recalculated, every time giving you a fresh set of defects to be analyzed.  Record the fraction of 'yes' values, and then repeat with different sample sizes.  Then you can chart the fraction of correct diagnoses against the sample size.

 

Remember, all of this depends on that first bullet, representing your actual results using a simulation.

 

Here is an example of what your simulation table might look like with a sample size of 20, assuming that one defect category is twice the size of all others:

 

    

 

Notice how clean the actual defect categories are here! I suspect this will not be so clear in your actual data, so you would need to use a different function in that column.

 

Here is the the simulation results table, here only using 100 rows (I recommend using 1000s):

    

 

You could record this in a table, and then repeat with other values to find your acceptable level of risk:

    

 

Here is a script to create the top table above:

View more...

New Table( "Monte Carlo Sample Size Example",
	Add Rows( 35000 ),
	New Script(
		"Distribution to Simulate",
		Distribution(
			Nominal Distribution( Column( :Defect Actual Category ) ),
			Nominal Distribution( Column( :Correct Category Identified ) )
		)
	),
	New Column( "Defect Actual Category",
		Numeric,
		"Ordinal",
		Format( "Best", 12 ),
		Formula( Max( Floor( Random Uniform( 0, 10 ) ), 1 ) )
	),
	New Column( "Sample Size",
		Numeric,
		"Continuous",
		Format( "Best", 12 ),
		Formula( 20 )
	),
	New Column( "Sampled Rows Sorted",
		Numeric,
		"Nominal",
		Format( "Best", 12 ),
		Formula( If( Row() <= :Sample Size[1], 1, 0 ) )
	),
	New Column( "Sampled Rows Shuffled",
		Numeric,
		"Nominal",
		Format( "Best", 12 ),
		Formula( :Sampled Rows Sorted[Col Shuffle()] )
	),
	New Column( "Sampled Defect Category",
		Numeric,
		"Ordinal",
		Format( "Best", 12 ),
		Formula( If( :Sampled Rows Shuffled == 1, :Defect Actual Category ) ),
		Set Display Width( 135 )
	),
	New Column( "Actual Defect Category",
		Numeric,
		"Nominal",
		Format( "Best", 12 ),
		Formula( If( Row() == 1, Mode( :Defect Actual Category << Get VAlues ) ) ),
		Set Display Width( 130 )
	),
	New Column( "Predicted Default Category",
		Numeric,
		"Nominal",
		Format( "Best", 12 ),
		Formula( If( Row() == 1, Mode( :Sampled Defect Category << Get Values ) ) )
	),
	New Column( "Correct Category Identified",
		Numeric,
		"Nominal",
		Format( "Best", 12 ),
		Formula(
			If( Row() == 1,
				:Actual Defect Category == :Predicted Default Category
			)
		),
		Value Labels( {0 = "No", 1 = "Yes"} ),
		Use Value Labels( 1 )
	)
);

If you have the computer power, you could skip a step and make the first 35k rows of your table use one sample frequency, the next 35k rows use another frequency, and so on.  Then you could simulate all values at once to populate the table in the last screenshot at once.