Sample size for failure detection with 0 failuresI have a population that I know the size of, let's say 50000. I have to manually check that values are correct. Now so far this has been 100% right after 14 samples, which took about 2 hours to do. Is there a way to either give the confidence of 0 failures at N samples, or how many samples I'd need to hit a specific alpha?
Re: Sample size for failure detection with 0 failuresYou can use the negative binomial distribution to derive sample sizes for these situations. A rough, but actually pretty accurate heuristic way to do it is to use the rule of 3. This rule says to show the failure rate is no greater than (1/n) with approximately 95% confidence and no observed failures, you would need to have 3*n samples with 0 failures. For example, if I want to be 95% confident that the failure rate is no greater than 5% (1/20, therefore n=20), I would need to observe 60 samples with 0 failures.
If you go the more exact route, the answer is 59 samples.
Re: Sample size for failure detection with 0 failuresHere's a really simple function that figures it out iteratively:
zeroFailBinom = function({fail_rate,alpha},
p = 1; //initial value
i=1;
while( p > alpha,
p = binomial probability(fail_rate, i, 0);
i++;
);
print("Required Samples: "||char(i-1))
);
zeroFailBinom(0.01,0.05)
