195338_en-US

liliconcom

DSDs and sparsity of effect principle

This is more of a conceptual question related to DSDs and their reliance on the sparsity of effects/pareto principle. I am trying to create a DoE for characterization of a bioprocess for manufacturing of drug product. The goal is to collect data for the parameters we've identified, create a model, and use the prediction profiler in JMP to set limits for each of the parameters so there are no "failures" in say, 10,000 runs. When selecting our parameters, we do a risk analysis of all the parameters in our process. So we began with about 40 parameters. Then we ordered them and ranked them based on risk of impact to the process outputs. After ranking them, we eliminated the bottom ~80% of parameters, leaving us with about 10 factors to test. What I am curious about, is whether it is appropriate still to assume the sparsity of effects principle that the DSD seems to rely on (If I'm understanding it correctly) in this situation since we already eliminated what we considered to be the bottom ~80% of parameters? Shouldn't a large portion of the remaining factors be significant then? Granted, out of the 40 parameters we began with, a lot of them were minor and we included them only to do our due dilligence.

 

My followup question is if we have budget to test ~48 conditions, would it be better to do a custom design and simply remove interaction effects I don't believe will be important until I get the N down to below 48? A DSD only requires 25 conditions for 10 continuous factors and since I don't have much characterization experience I'm a little worried it won't provide enough data to confidently model where we should put our process limits if any of the parameter ranges we have decided to test lead to results that fall outside our output specifications. I realize we could always do a DSD and then supplement if we need more runs, but this would be less efficient for us (since it would require more blocks) than just starting with a design that counts for 48 runs. 

 

Thanks for your help!

Re: DSDs and sparsity of effect principle

In the first paragraph of your opening remarks it sounds to me like you are concerned that, wrt to the 10 factors, a majority of them will be active. What does your process and domain expertise tell you? Do you have pre-existing process data that you can leverage using data visualization, exploratory data analysis, and simple to advanced modeling techniques to help inform your knowledge? If you truly have ZERO domain/process knowledge then I think you should go the DSD route as a FIRST experiment to build that knowledge. From there, with remaining resources, work on the optimization part of your problem using a SECOND experiment. In practice the new product/process development teams I worked on found the sequential DOE process to be the most efficient to solving the practical problem. DSDs are first and foremost SCREENING designs. 

Re: DSDs and sparsity of effect principle

First, thank you for your reply. Unfortunately, while I have a good feel for which of these parameters will have an impact on process outputs we constantly measure such as pH, cell growth, etc., I'm less certain of their impact to the drug product (and they could have an impact, which is why they scored higher in our risk assessment) and that's typically something we just have to investigate. We don't have that data for this specific process. So based on your reply it sounds like in this scenario your recommendation is to start with a DSD and then fill out the significant effects with more runs. More specifically, I would run the DSD, and then use the fit model tool to one by one eliminate effects not statistically significant (cutoff around ~p=0.05 or any clear cutoff around there). Then I would take the remaining effects and put them in the custom designer with the remaining runs we have budgeted to get the best design possible. Does that sound similar to how you would do it? Thanks for your help! 

Re: DSDs and sparsity of effect principle

The idea of screening is to separate the "trivial many from the vital few."

 

I would consider adding a few more runs above the minimum number of runs in your DSD. JMP now defaults to 4 extra runs. You might consider adding 1 or 2 more runs. This small increase will dramatically improve the power of your design.

 

No, do not take your knowledge from the DSD or any other initial experiment and start a subsequent experiment with Custom Design. Use the Augment Design platform. This platform uses the custom design algorithm but it also uses your existing runs. So you can incrementally improve the power and predictive performance of a design. This approach is known as sequential experimentation, which has been advocated for almost five decades.

 

(By the way, the DSD is a special case of an alias-optimal custom design.)

Re: DSDs and sparsity of effect principle

I recommend using the Fit Definitive Screening platform or the Generalized Regression (JMP Pro) instead of Fit Least Squares.

Re: DSDs and sparsity of effect principle

Hi @markbailey, thanks for your help. I have played with the augment design tool a little before. I would like some more clarity on how to differentiate the vital few from the trivial many though when I use the augment design. When you first press on "Augment Design", a screen comes up that asks you to select your response and factor parameters. Let's say that only 4 out of the 10 variables come up as significant from the initial DSD experiment. Should I only select the 4 factors in this screen? Or should I select all 10, and then on the next screen when it asks for model terms I should start removing terms that aren't significant, and including interaction effects and polynomial terms that I am interested in. Or do these do the same thing?

 

Thank you

Re: DSDs and sparsity of effect principle

You only carry forward the factors that you decide are active. That decision might not be clear about all the original factors.

 

Yes, you should modify the model terms to update them based on your current knowledge or questions.

 

I recommend Help > Books > Design of Experiments. There are chapters about screening, DSD, and augmenting designs.

Re: DSDs and sparsity of effect principle

Thanks @markbailey . One more quick question on a different topic. Let's say I have 3 factors, temperature, pH, and production length. I want to see the impact on drug product produced. What would be the negatives of designing an experiment with only 2 factors (temperature and pH) and taking different time point samples to represent production length. Let's say you have 3 levels for each factor (and we'll assume full factorial). So 3 x 3 is 9 runs. For those 9 runs you take 3 time points each. Well when you plug this into JMP software you make it look like actually you had 27 runs, when in reality a lot of those different time points were the same condition just at different times. Are you somehow biasing the results for the other 2 factors (temp and pH) doing it like this? Or is this acceptable and recognized as making experimentation more efficient?

Re: DSDs and sparsity of effect principle

It all depends on how you view production length. Is it a factor (input) to influence the outcome or is it a repeated measures design? Would you select a level when optimizing or determine a window in the design space for it? That is a factor.

 

Are you interested in the time course? If so, then you can use JMP Pro Functional Data Explorer, although three points is not much of a function. Or you could use another model for the Y( time ) and use model parameters for responses if you have JMP. Either way, it is part of the response.

 

Bias is a model thing, not a design thing. (Of course, the design must support any of the models in question.)

 

Efficiency is a model thing, economy is a design thing.

Re: DSDs and sparsity of effect principle

@markbailey  It's definitely a factor. Yes we would be trying to define process limits to it (determining a window). So you're saying in that case it would need to be it's own factor in the design, and it would be improper to not include it in the design and simply take time course samples. But it's not clear to me why that is. What's the issue with taking time course samples, and then post-experimentation simply adding the data back in as if it were its own factor? In my example with the 3 factors, this would look like copying and pasting your 3 x 3 design two more times, and adding a new time column where you have t1, t2, and t3 for each set of 3 x 3 conditions. Then filling out the response column with your results, where rows with t1 have the results from your first timepoint, and rows with t2 have results from your next timepoint, etc. Is it because when you fit a model there's an assumption of independence between your data points for any given factor? So when you take a timecourse your t3 sample is not independent of your t2? 

Re: DSDs and sparsity of effect principle

Are the 10 factors you have chosen to experiment on all from the same process step.  Typically, biopharma processes are multistep (cell lysis, clarification/TFF, refold, chromatographic purification, buffer exchange, ...).  If the 10 factors are dispersed across several steps, then one nice advantage a DSD has is that these project into response surface designs for any 3 factors, and even if there are 4 factors you can build a nice prediction model.  Also, in these process the factors from one step don't directly interact with the factors from another step.  The can interact "indirectly", as the output from 1 step (which may be dependent on factors from that step) is an input to another step.  This "indirect" interaction is often smaller than any process factor effect, however.  

 

Let's say that step 1 has 3 factors, step 2 has 3 factors, and step 3 has 4 factors.  Then you can run the DSD on the full process (Steps 1-3), characterize  and measure the product quality at each step, and build many different models.

A. A model to predict product quality after Step 1

B. A model to predict product quality after Step 2.  You can use the resulting quality measurements from Step 1 as covariates in this model.

C. A model to predict product quality after Step 3.  You can use the resulting quality measurements from Step 1 or Step 2 as covariates in this model.

 

Don't forgot to consider blocking when doing this type of work.  Often the block is associated with the analytical run used to measure the material coming out of the experiment, and you can block the experiments on "assay run" to account for any potential bias or confounding that will come from the assay.

 

All of that being said, I would definitely budget for follow on experiments, as often the DSD doesn't give you all the information you want about the process, or you may discover that you didn't choose the best ranges for experimentation, etc...

 

 

Re: DSDs and sparsity of effect principle

Please do not 'put words in my mouth.' I did not say that your idea of treating time as part of the response is "improper." It is simply not what I would do. Since there is no 'what,' there is no 'why.'

 

You think in terms of combinations. I would say that you are design-oriented. I think in terms of estimation. I say that I am model-oriented. Both orientations work and one way does not have to explain itself or justify itself to the other way. (I work in a group that approaches experiments from both directions.) There is nothing that you have proposed that won't work as long as the design of the experiment, execution of the experimenta runs, and the data analysis (model) are consistent. I start with the model. You start with the design.

 

The issue of independence is real. You are proposing to run a repeated measures experiment but analyze it as a completely randomized factorial design. A statistical model has some combination of fixed and random effects that should reflect how the data was generated as well as what you want to know about the response. You are creating a series of whole plots with the two fixed factors in which to observe the third factor. The 3 observations in each plot are correlated this way.

Re: DSDs and sparsity of effect principle

Thank you @MathStatChem . Unfortunately no it's 10 factors for the same step of process (production with cell culture). I definitely don't believe all 10 will be important, but it could be somethingl like 4-6 (although mayb even less). We just aren't sure because we don't have a whole lot of data with this process. 

Re: DSDs and sparsity of effect principle

@markbailey  thanks for your help. I wasn't trying to put words in your mouth, just trying to get clarity around what you're saying. I think I understand now. 

Re: DSDs and sparsity of effect principle

Be careful.

 

It is more than potential interactions among factors across more than one processing step that should be considered. It is likely that the early steps represent a restriction on the randomization of the runs that involve the factors in the later steps. If that is the case, then a DSD is not appropriate. A custom design with appropriately defined hard to change and very hard to change factors is likely to be more realistic and effective.

 

Improperly considering the multiple levels of randomization and ignoring the additional random effects in the model can compromise the inferential information from the analysis.

 

I apologize if I misunderstood your suggestion.

Re: DSDs and sparsity of effect principle

That's a good point.  If you are not going to carry material all the way through the steps, then for that number of factors, you might be better off to just characterize each step separately.  

 

I've done it both ways.  

Re: DSDs and sparsity of effect principle

I'm guessing that you already have a good idea on what is important and what interactions may be present.  

 

If you haven't already, it might be good to make an "interaction" map.  Just go through each pair of factors and assess if you think they will interact.  you can also consider if there will be an max or min output result in the middle of the design range.  If that is true, then likely you will also have at least one quadratic effect.  Then count up the main effects (10 in this case) and the number of potentially important interactions and quadratic effects, and make sure that number is less than the number of runs in the experiment.  You can also check to see if the two factor interactions are strongly correlated with each other, and if so you might do some shuffling of the assignment of the factors to the DOE columns.  Another advantage of adding the +4 additional runs in this case (10 continous factors) is that it creates a design with less confounding among the two-factor interactions and quadractic effects.  

Re: DSDs and sparsity of effect principle

I'm replying here after reading all of  @markbailey and @MathStatChem 's input. I concur with everything they are advising or discussing. My recommendation was 'go slowly'. Start with a 'first' experiment...and I think we are all suggesting a DSD if that works logistically for you, as well as from an experimental design point of view. What I'm suggesting for after the DSD and it's results (statistical and practical) is...Pause. Take a deep breath with your team. Ask the following questions:

 

1. What have we learned wrt to solving the practical problem at hand?

2. Given what we've learned, what is the next logical step to solve the problem?

 

The answers to 1. can be multiple, conflicting, or just plain useless (but even then, one of the sage engineers I worked with when an experiment showed 'nothing' informed us all, "Hey, knowing an is 'not' can be just as valuable as knowing an 'is'." Try not to get too wrapped up in picking a specific DOE method (optimal design, classic RSM, more screening) at this point. Instead fit the problem with this hard won 'new' knowledge to the optimal next analytics steps. I think this is the essence of Mark's comments. DOE or otherwise. For example, what if you learn that your measurement system for the 'drug effect', that seems to be paramount here, is extremely noisy, making it hard to detect a signal above the noise? Subsequent DOE work could still be problematic...maybe it's resources better spent on the measurement system first? Or what if some factors and the levels you've chosen are persnickety? 'Persnickety' is a technical term I use when factors in an experiment are difficult to control, set, or have lots of known and unknown sources of variablity. In working with my biotech JMP process/product design customers over the years...any time they worked with 'cells'...well 'persnickety' was being polite. 

Re: DSDs and sparsity of effect principle

Thank you. I think I have a good idea of the path forward now, and will plan to use a DSD as the first run.