Determining sample size for categorical factors

Question

I'm trying to determine the sample size required to make a go/no-go decision on whether or not to make a process change for a categorical factor with 2 levels, and both categorical and continuous responses.  We ran some experiments and saw no statistical significance in the responses between the two levels of the categorical factor, but now we want to know how confident we are that there is no difference.  To answer this, I started playing around with the JMP sample size and power calculator for two sample means.   I originally thought my sample size was 432, because there were three experiments which each consisting of 72 samples treated with the control process and 72 samples treated with the experimental process, (432 = 72*2*3), but then I wasn't sure what to input as my standard deviation for the categorical response.  I found one resource on the internet that suggested I could calcualte a coefficient of unalikeability to measure variability for categorical responses:   http://jse.amstat.org/v15n2/kader.pdf If I use this approach I get that at a significante leve of 0.05 with 428 samples and a coefficient of unalikeability of 0.3 that I'll have an 87% chance of detecting a significant difference in the categorical response of 0.09%.

Anyone have any thoughts on this approach, or ideas on how to figure out sample size and confidence in the results another way?

A colleuge of mine suggests we say the sample size here is only 3, for 3 runs consisting of 72 samples treated with the control process and 72 samples treated with the experimental process.

Thanks,

Jolene

Jolene · Answer

Hi Georg, I have a two step process, and I varied the treatment of my samples in step 1.  The conditions were treatment A (control) vs treatment B (alternative).  I had 216 samples that went through treatment A, and 216 samples that went through treatment B.

All of these samples then went through the second process.  Due to constraints I had to process them in three separate runs, each consisting of 72 samples from treatment A and 72 from treatment B.  Runs 1, 2, and 3 were processed similarly.

I am now analyzing the continuous and categorical responses to this categorical factor of treatment A vs B.  I found no significant difference in the responses, but my collegues asked if the sample size was large enough to feel confident in making the decision to go ahead and change the process.

Thanks!

Kevin_Anderson · Answer

Hi, Jolene!

Retrospective power analysis is generally an inappropriate thing to do.  As @Georg implies, we know very little about your experiment.  You designed it (without considering the power of the experiment?), ran it, collected the data, did an analysis, and did not achieve “significance.”  So now you compute power retrospectively to see if the test was powerful enough or not.  That is a specious question.  We already know it wasn’t powerful enough to detect the effect size...that’s why the result isn’t significant!

Power calculations are useful prospectively for design, not retrospectively for analysis.  You might wish to entertain modifying your hypothesis for a subsequent experiment to be one of equivalence or noninferiority; Please see the enclosed Hoenig and Heisey paper.

Without any understanding about your experiment's objectives, design, conduct, or analysis, the sample size seems quite large.  You may be able to use the data from your experiment to formulate a simulation model to explore the prospective power for specific potential designs and analyses for a subsequent experiment.  It seems to me that this approach would do the best job of answering your questions.

Good luck!

Hoenig Heisey The Abuse of Power The Pervasive Fallacy of Power as Data Analysis TheAmStat2001.pdf

Georg · Answer

First it is important to understand your experiment.

So you have 72 different samples, and you can do different experiments on each sample?

After you can do a paired comparison of different experiments on each sample?

You want to compare your old process to a new process with two different categorical settings, so for a full run we would have 3 treatments x 72 samples?