Sample size for failure detection with 0 failures

Question

I have a population that I know the size of, let's say 50000.  I have to manually check that values are correct.  Now so far this has been 100% right after 14 samples, which took about 2 hours to do.  Is there a way to either give the confidence of 0 failures at N samples, or how many samples I'd need to hit a specific alpha?

cwillden · Accepted Answer

You can use the hypergeometric distribution to derive sample sizes for these situations (finite population sizes), or the binomial for infinite population sizes.  At 50,000 units, they will effectively give the same answer.

A rough, but actually pretty accurate heuristic way to do it is to use the rule of 3.  This rule says to show the failure rate is no greater than (1/n) with approximately 95% confidence and no observed failures, you would need to have 3*n samples with 0 failures.  For example, if I want to be 95% confident that the failure rate is no greater than 5% (1/20, therefore n=20), I would need to observe 60 samples with 0 failures.

If you go the more exact route, the answer is 59 samples.

cwillden · Accepted Answer

Here's a really simple function that figures it out iteratively:

zeroFailBinom = function({fail_rate,alpha},
	p = 1; //initial value
	i=1;
	while( p > alpha,
		p = binomial probability(fail_rate, i, 0);
		i++;
	);

print("Required Samples: "||char(i-1))
);

zeroFailBinom(0.01,0.05)