Testing - Sample sizes for a full factorial

Liz_Perkin

 

Hi,

I’d really like some help with sample sizes for a full factorial design for a Marketing campaign I’m involved in - looking at response rate. I want to test 4 things, all have 2 levels, cannot be changed and are categorical. If I were to conduct 4 separate A/B tests the cell volumes would look like this:

 

Control response rate = 0.6%
		Success	alpha	Power	Sides	Volume/cell	Total volume
	Test 1 (A/B)	50% uplift	0.8	0.2	1	4687	9374
	Test 2 (A/B)	10% uplift	0.8	0.2	2	156606	313212
	Test 3 (A/B)	10% uplift	0.8	0.2	2	156606	313212
	Test 4 (A/B)	10% uplift	0.8	0.2	2	156606	313212
						Total:	949010

 

 

But I’m keen to look at the interaction effects (only at 2* level) between the factors. I would like to set up a 16 cell full factorial DOE but am unsure of what sample sizes to use in each of the 16 cells. Any help would be greatly appreciated.

Let me know if any more information is required

Thanks, LP

malcolm_moore1

Hi Liz,

 

The procedure for determining power estimates in JMP Pro for a DOE with a binomial response as in your case has a number of steps as follows:

 

• Design your experiment in the normal way and select simulate responses from the DOE hot spot and specify the coefficients for your simulated model, e.g. I used the following coefficients to explore effects equal to the size of random noise, 1.5 times the size of random noise, 2 times the size of random noise and so on (see screen shot attached)
• Click make table which gives one simulated scenario based on 100 trials in the case of the example provided (or asking 100 people).
• Run a generalised regression model using binomial response and model including main effects and 2-factor interaction effects (see scripts in attached table called power analysis with binomila response)
• In the GenReg report, right-mouse click over the p-value column and select simulation.
• In the resulting simulation table run the power analysis script and examine the lower 95% confidence interval under simulated power for each term in your model. This gives you the minimum anticipated power for each effect in your model for various alpha rates (type I error or significance level). An example is attached in the table called Generalized Regression Simulate Results (Prob > ChiSquare) n trials = 500
• You may need to run the power scenario using several different number of trials for each row in your design matrix. This is easily done by copying and editing the column formula in the first table. For illustration purposes, I’ve given scenarios for n=100, n=250 and n=500 trials (or people surveyed) per row in the DOE matrix.
• Using the power for each of these scenarios you can make a decision about the number of trials (people surveyed) per row in your experimental design to give acceptable power and type I error trade-off.

 

Best,

 

Malcolm

Example Simulation.jpeg

Generalized Regression Simulate Results (Prob > ChiSquare) n trials = 500.jmp

Power Analysis with Binomial Response.jmp