Are Custom Designs OK for Variance Components ?

Question

I have a statistics question:  What are the pros and cons of using a Custom optimal design to estimate variance components?

For example, I have four nominal factors with three to four levels each.  All factors will be treated as random and I will estimate REML variance components for all factors.   (I'm not interested in tests of significance or fitting a fixed effect model.)  There are too many level combinations to use a full design.  Is using a custom design to select a small number of level combinations to run OK for this type of analysis?

Thanks!

Paul_J · Answer

Mark, Thanks.  I was a bit unsure if the partial correlation of effects in a custom design would be a problem for the variance component estimates, but good to hear the optimal design would be OK.  I had been considering a Latin or Youden square, or BIBD to balance the number of times factor level combinations were observed.

As for the design being too small, yes I do have that concern also.  My main metric is degrees of freedom, but would be interested if anyone knows of another metric.

Regards,

Paul_J

Mark_Bailey · Answer

Yes, custom design is OK for collecting data to estimate variance components. In fact, any design method is OK. It might not be optimal, but it will be OK. (I honestly do not have any information about designs that are optimal for estimating the response variance.)

Your comment, "There are too many level combinations to use a full design," though, raises the main concern. Small designs are efficient and effective for estimating the mean response (fit model of the mean) when the effects are large and the standard deviation of the response is small. You need much more data to estimate the variance as well as the mean model. That is a big difference between the estimation of fixed effects and random effects. All the design methods that I know are for the fixed effects model.

If you have only a few samples ("I have four nominal factors with three to four levels each.") then your variance estimates will have very wide confidence intervals. Often the interval will include zero even when there is a non-zero contribution from a factor. The fact is, you must have many more observations in order to obtain a stable and precise estimate of the spread.