40496_en-US

liliconcom

Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

Hi all,

 

Our internal practice for running precision studies on an assay will produce multiple CVs for different variance components (e.g., Day-to-Day, Weighing-to-Weighing, etc.). Due to this, the question of “can we use these CVs to determine statistical power in future experimental designs?” often arises.

 

The answer:

 

However, it’s not easy. Marianne (MTOF) has developed a workflow to take in an experimental design, integrate the design with the assay’s CVs, and simulate running the experiment 1,000 times. From this, we determine how many times each fixed effect was determined to be statistically significant in order to calculate power for that effect.

 

Unfortunately, performing this process manually is fairly time intensive (and very easy to make mistakes). As such, I have worked on automating it.

 

The Add-In I have attached is still a prototype, but MTOF and I have played around with it and believe that it should be ready for testing.

 

So, how does it work?

 

Let’s say that I run a precision study on a new assay. In this assay, we determine that there is a Day-to-Day CV of 4%, a Sampling-to-Sampling CV of 3%, and a Measurement-to-Measurement CV of 2%.

 

After a while, a second machine is purchased. We want to design a quick, small study to see if the original machine and the new machine produce similar results.

 

We create the following simple experimental design, and want to know if it's "good enough":

(“In one day, test two samplings with 2 measurements per sampling on each machine”)

 

Now let’s get started:

1. Start the Add-In
2. A window will appear showing all currently open data tables:
   1. 
   2. If the experimental design of interest is currently open, select it from the menu and click OK.
   3. If the file is not open, select “Import New File” and then hit OK. Use Explorer to select the file of interest.
3. The main interface will launch, allowing you to define the model that should be used:
   1.  
4. For the demo table shown above, the only fixed effect is “Machine”, so I will add it to the model
   1.  
5. My random effects are Day, Sampling, and Measurement. First I add the Day and Sampling effect:
   1.  
6. Due to how my table was set up, Day and Sampling both need to have their nesting set correctly:
   1.  
7. Now use the “Toggle Selected as Random” button to set Day and Sampling to random effects:
   1.  
8. Two boxes will appear in the “Define Random Effect CVs” sidebar to allow you to set the CVs for these random effects. I’ll put in 4% for Day and 3% for Sampling:
   1. 
9. Now, we also have the “measurement-to-measurement” CV to worry about, and this column wasn’t added to the input data table. Since Measurement is the lowest level random effect, it would be confounded with the residual. So, we can use the “Create New Residual Column” to create a column for measurement error, and set its CV to 2%:
   1.  
10. Next, select which column is the response column (“Result”) and then click OK
    1.  
11. A new datatable will be created based on the input datatable, and “_PowerSim#” will be amended to the end of the filename. As such, no harm will come to your original file.
12. A “Run Simulation” script will be embedded in the new file. Click the icon to start an evaluation. 
    1.  
13. First, select the fixed effect where you would like to detect a difference. For this design, we only have one choice (Machine):
    1.  
14. A box will appear within the “Difference To Detect” sidebar. By default, everything will be set to 100, which corresponds to the mean.
15. Let’s say I want to know if my experimental design will have the power to detect a 5% difference in level dude to machine. To get this, I change the value for the New Machine to 105:
    1.  
16. By default, the critical alpha value is set to 5%. If desired, this can be changed in the “Crit Value” box. I’m happy with 5%, so I’ll hit OK.
17. Now we wait for the models to all run. In the future, we’ll try to speed this step up, but for now it takes a minute or so to run.
18. When it’s done, a pop-up will appear with the results:
    1.  
19. As a result, the power to detect a 5% difference in response at an alpha of 0.05 due to the effect of “Machine” is 23.6%. So, not a great design. You can go back and adjust the difference to detect and/or the alpha value used, but it’s probably better to make a more robust design.
20. Now this was a pretty simple example. More complicated models and examinations can of course be utilized:
1.    

There are certain features that are not implemented (nested fixed effects, interactions other that Nominal*Continuous), and work that can be done to possibly increase speed (e.g., moving to an iterative system instead of printing all data within the data table up front).

 

Please let Marianne and I know if you have any questions.

 

EDIT: Notes on version 3 can be found here

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

I've attached version 3 of this add-in to the first post.

 

Beyond the typical bug fixes, there are two notable changes in functionality.

 

Firstly, the tool no longer duplicates the experimental design one thousand times (resulting in the datasheet being 1000×[rows of design] rows large). Doing so was helpful in bugtesting, but we've switched to an iterative method where instead of running 1000 models at once, it now runs one model, logs the result, reconfigures the random contribution columns, and then runs the next model.

 

This doesn't really offer much in the way of speeding up the process, but what it does is prevent the hangs/freezes encountered when very large experimental designs were used.

 

Secondly, the tool now has an option to define "Error Treatment":

 

 

The "Normal Error" treatment uses the same Normally distributed error as the previous versions. Selecting "Microbial Plate Error" does things a little differently: (1) the error contribution columns will have a Lognormal distribution instead of Normal, (2) the response will be Log10 transformed (so no need to select "Log Transform Response" in the "Select Response Column" outline box), (3) variance will need to be defined in terms of standard deviation instead of CV, and (4) user will be required to specify the response level when running a new simulation:

 

 

This functionality was introduced to better account for microbial plate data. The methodology behind this methodology is still in active development, and the tool's implementation has been checked but not fully validated, so take any results from using this error treatment with a grain of salt. If there's any worry, I recommend to continuing using "Normal Error" alongside a Log Transformed Response for plate data as before.

 

 

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

>One instructor had even told us that the randomized block design for experiments was originally concieved specifically for plant science in order to help handle the many issues with different levels of soil quality, water availablity, etc., in field trials (the "blocks" were literal physical square miniplots of crops in the fields). Not sure how much truth there is to this, though

 

Where do you think terms like split plot came from?  Fisher designed experiments for agronomic field trials and came up with ANOVA. And we definitely encourage our cooperators to arrange the randomized block designs along 'similar' chunks of field.  So i'd believe your wise instructor. 

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

But these are still just general considerations - if you want to know which setup will be most cost-efficient and give the best power, then use Adams detailed calculation description or add-in

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

@jsdejmp

Adding more replicates at the higher level IS actually also more effective if you don't maintain the number of replicates at the lower level the same but as in your example you keep the same total (as long as the setup is nested as earlier mentioned).

I can tell for sure that 10x5 is better than 5x10 regardless of the CV's! If it is nested, when you count the total 10x5 gives 10 trials and 50 plots, while 5x10 gives 10 trials and 50 plots. So the plot-to-plot variation is in both cases reduced with a factor 50, while the trial-to-trial variation is reduced with a factor 10 or a factor 5. 

If it is not nested, i.e. the trial is run in the same place 5 times, so it is the same 10 plots you reuse over and over, then it is a different situation, but I don't imagine this is how you run field trials? 

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

Ah, that makes more sense -- I'm guessing he said the correct thing and I just misrembered

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

 @ABI

I think your instructor was referring to the split-plot design - that was developed for agronomy field trials - and is where the name comes from.

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

Happy to help in any way I can.

 

Note that we use a log transformation for plate data because the error is never normally distributed, and the transformation is necessary to approximate a normal distribution.

 

As an aside, I definitely feel your pain on field trial issues. My undergrad and graduate work was all on plant genomics, so I have fought that fight (just wait until someone asks you to combine Greenhouse results with Field Trial results -- just say no!). One instructor had even told us that the randomized block design for experiments was originally concieved specifically for plant science in order to help handle the many issues with different levels of soil quality, water availablity, etc., in field trials (the "blocks" were literal physical square miniplots of crops in the fields). Not sure how much truth there is to this, though

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

@ABI, thank you for this well put together explanation and guide on how you approach designing your experiments. I definitely think I can apply this system to our experimental approach. We've been working on getting good estimates on our CVs, but as you said, we really first need to understand how our various random effects interact. I think we need to do additional modeling to capture some of this info.

 

(Btw thank you for mentioning why a log transform is helpful when capturing CV; I was just speaking with MSQM about that on a QC SOP and I wasn't sure on why that was).

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

Hi @MTOF,

 

Reply to #1 - Adding more at the higher level is more effective only if we maintain the number of replicates at the lower. Considering a budget constraint, the question is closer to "Should we do 5 trials x 10 reps or 10 trials x 5 reps". Adding to this practical question is a more abstract and highly debated one: does an individual trial constitute an entire experiment or just a single data point? As no one should draw any treatment conclusions from a single trial, yet clearly there is considerable design and operational cost for this one experiment and its data ought have appropriate weight. (The rabbit hole goes deeper still, as different crops have different field sensitivities and therefore different CVs and therefore a different optimal number of trials and reps; and we're also currently interested if different subcontractors who execute the trials might also have different levels of ability to manage a field and measure responses. Not to mention weather and year-over-year variance at the same location. It's easy to develop an attitude of surrender when mother nature easily thwarts our hard work.)

 

Reply to #2 - Thank you for clarifying this more.  I wasn't sure i could approach it like this.

 

 

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

Hi Jameel

 

An addition to Adams reply on your question "is there a quick way to identify which aspect of a design needs more samples":

 

1. Adding replicates at a 'higher' level is always more effective than adding them at a lower level (but typically also more expensive).

 

This is because replicates at the upper level also gives replicates at all lower levels. At least with a hierarchical/nested setup as we have made this tool for and which I also assume you have in field trials. For example if currently our setup is 2 days x 2 weighings x 3 plates (total 2 days, 4 weighings and 12 plates) adding 1 day gives a total of 3 days, 6 weighings, 18 plates.

 

2. The relationship between the CV's tell you where it is most 'necessary' to add replicates.

 

If your trial-to-trial CV is much larger than the within-trial CV, then it necessary to add more trials (adding more replicates within each trial will contribute very little). If the trial-to-trial CV is much smaller than the within-trial CV, then it would still be effective to add more trials as you would also get more replicates (plots of land or plants or whatever your replicate is), but probably much cheaper and just as effective to add replicates within the trial.

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

Hi Jameel,

 

Yes, you can do this (perhaps without this particular tool, even). If you have the random effect variance components for an assay (from JMP or otherwise), those component values can estimate the CV of a particular assay quality setup.

 

How the CV components (i.e., effects) interact with one another is likely more important than their percent value. For your example, more locations are probably more important than more reps. In fact, it might be possible that the within-location reps aren't true replications at all, and they're more close to sample repeats than actual replicated measurments (and therefore less useful -- but, without knowing more about your experiment, I can't really speak to this).

 

So let's work through an example that I deal with alot: micro plate assays.  Say we have three CV components: the random effects of day-to-day variation, weighing-to-weighing variation, and plating-to-plating variation.

 

In our assay design, we have these three CV components with the following nesting:

• Day&Random
• Weighing[Day]&Random
• Plate[Day,Weighing]&Random

So each day of plating gets X amounts of weighings (subsamplings), and every weighing gets Y amount of plates.

 

Next, after performing a study to determine precision, you would get the variance component values from the REML table and their corresponding CVs (i.e., if the response was log transformed, the CV of the variance component is equal to the "Sqrt Variance Component" field in the REML table. If not transformed, it is equal to the sqrt variance component from the REML table/Mean of Response from Summary of Fit table).

 

So let's say that we have a Day-to-Day CV of 5% (var component of 0.0025), a Weighing-to-Weighing CV of 10% (var comp of 0.01), and a Plating-to-Plating CV of 8% (var component of 0.0064).

 

If we add up all three components, we get a total variance component value of 0.0189, which translates into a CV of 13.7% (sqrt[0.0189]) if only one day, one weighing, and one plate are made.  

 

Due to the nesting strategy outlined above, calculating a predicted population CV for a quality setup in this assay would follow the following calculation:

 

 

 

Then in Excel it's a simple matter of making this into a formula and filling down to try out different combinations, or do it in JMP by making a formula column.

 

Example:

 

  

Above we have three designs that each only have 5 total plates. You can see that having a 5day×1weighing×1plate setup results in a much lower CV than a 1day×1weighing×5plate design, dispite Day-to-Day variation having the lowest component CV.

 

And then you can play around with other designs to figure out what is realistically best:

 

 

 

So here's three designs, each with 20 plates and fairly close CV values. It then becomes a conversation of which setup is most feasible/cost appropriate. You could also use the Solver tool in Excel with some rules to help you if it's going to be a bunch of guess-and-checking.

 

Regardless, as a result you will be able to evaluate whether more locations or more reps within a location are valuable, and by approximately how much (i.e., what is the change in CV between adding another location, or adding ten more reps to each already existing location?).

 

Now, this particular tool was built because micro plate assays are always imprecise, and a lot of micro studies have the desired goal of stating that "there is no difference between X and Y" (i.e., there is no difference between agar type X and agar type Y). However, if the experiment lacked sufficient power to actually find a statistically significant difference, we could end up concluding that an alteration to an assay (switching agar types) has no effect when it actually does. Which is a very bad thing.

 

However, if false negatives are not as big of a concern with your experiment, then you might not need this particular tool. If you do think you'll need it, let me know -- it needs some optimization so that it doesn't cause JMP to hang on very large datasets, so if you're working with a large dataset I'll prioritize this type of optimization first

 

-Adam

 

 

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

@ABI, @MTOF, using the values you provide in your examples and the tool, is there a quick way to identify which aspect of a design needs more samples? For example, in the first example given, would adding more measurements across more days improve power more than adding measurements within a day (since CV is 4% vs 3%)? Does the effect with higher CV indicate thats where you should load more measurements?

 

For agronomic field trials, where we might have 10 trials with 5 replicates of each treatment within the trial, we can calculate overall power using the overall error from a set of data. For a JMP style power analysis, additional data - regardless of where it comes from, more locations or more reps – is treated equally. However, reality is that there are differences in site-to-site variability and within-site (rep) variability.  We can have more locations ($$) or more reps within a location ($). I personally favor more locations because more reps in one field would only give us greater confidence in that one field; more locations allows us to be ‘more sure’ observed treatment differences are not overly weighed by any one place. Can your tools help me sort out this issue?

 

I am imagining creating more intelligent algorithms for experimental design that weigh location-to-location and the within-location variance and optimize it within a specific budget. For example, "We have $X to run Jumpstart on corn...whats the best way to distribute the number of locations and reps?"

Re: Power Simulator Tool (Determining power for an assay with existing Precision Study CVs)

Hi all,

 

I've fixed a bug that would cause a crash with loading in very large experimental designs and attached it to the first post. Please note that the current methodology employed is not optimized for speed yet, and running the 1000 simulations will take a very long time on large designs (sometimes even causing JMP to hang/freeze if they are large enough). I have some ideas on how to speed the tool up, but MTOF and I have some ideas on how to improve the applicablity of the distributions used for our random effects, as well as a general overhaul of how Novozymes approaches plate-based data (which is the primary driver for this tool's existance). As such, functionality improvements will be prioritized over speed in the immediate future.

 

Thanks!