nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

DSDs and sparsity of effect principle

dbio27

This is more of a conceptual question related to DSDs and their reliance on the sparsity of effects/pareto principle. I am trying to create a DoE for characterization of a bioprocess for manufacturing of drug product. The goal is to collect data for the parameters we've identified, create a model, and use the prediction profiler in JMP to set limits for each of the parameters so there are no "failures" in say, 10,000 runs. When selecting our parameters, we do a risk analysis of all the parameters in our process. So we began with about 40 parameters. Then we ordered them and ranked them based on risk of impact to the process outputs. After ranking them, we eliminated the bottom ~80% of parameters, leaving us with about 10 factors to test. What I am curious about, is whether it is appropriate still to assume the sparsity of effects principle that the DSD seems to rely on (If I'm understanding it correctly) in this situation since we already eliminated what we considered to be the bottom ~80% of parameters? Shouldn't a large portion of the remaining factors be significant then? Granted, out of the 40 parameters we began with, a lot of them were minor and we included them only to do our due dilligence.

My followup question is if we have budget to test ~48 conditions, would it be better to do a custom design and simply remove interaction effects I don't believe will be important until I get the N down to below 48? A DSD only requires 25 conditions for 10 continuous factors and since I don't have much characterization experience I'm a little worried it won't provide enough data to confidently model where we should put our process limits if any of the parameter ranges we have decided to test lead to results that fall outside our output specifications. I realize we could always do a DSD and then supplement if we need more runs, but this would be less efficient for us (since it would require more blocks) than just starting with a design that counts for 48 runs.

Thanks for your help!

Accepted answers

All comments

P_Bartell

In the first paragraph of your opening remarks it sounds to me like you are concerned that, wrt to the 10 factors, a majority of them will be active. What does your process and domain expertise tell you? Do you have pre-existing process data that you can leverage using data visualization, exploratory data analysis, and simple to advanced modeling techniques to help inform your knowledge? If you truly have ZERO domain/process knowledge then I think you should go the DSD route as a FIRST experiment to build that knowledge. From there, with remaining resources, work on the optimization part of your problem using a SECOND experiment. In practice the new product/process development teams I worked on found the sequential DOE process to be the most efficient to solving the practical problem. DSDs are first and foremost SCREENING designs.

dbio27

First, thank you for your reply. Unfortunately, while I have a good feel for which of these parameters will have an impact on process outputs we constantly measure such as pH, cell growth, etc., I'm less certain of their impact to the drug product (and they could have an impact, which is why they scored higher in our risk assessment) and that's typically something we just have to investigate. We don't have that data for this specific process. So based on your reply it sounds like in this scenario your recommendation is to start with a DSD and then fill out the significant effects with more runs. More specifically, I would run the DSD, and then use the fit model tool to one by one eliminate effects not statistically significant (cutoff around ~p=0.05 or any clear cutoff around there). Then I would take the remaining effects and put them in the custom designer with the remaining runs we have budgeted to get the best design possible. Does that sound similar to how you would do it? Thanks for your help!

Mark_Bailey

The idea of screening is to separate the "trivial many from the vital few."

I would consider adding a few more runs above the minimum number of runs in your DSD. JMP now defaults to 4 extra runs. You might consider adding 1 or 2 more runs. This small increase will dramatically improve the power of your design.

No, do not take your knowledge from the DSD or any other initial experiment and start a subsequent experiment with Custom Design. Use the Augment Design platform. This platform uses the custom design algorithm but it also uses your existing runs. So you can incrementally improve the power and predictive performance of a design. This approach is known as sequential experimentation, which has been advocated for almost five decades.

(By the way, the DSD is a special case of an alias-optimal custom design.)

Mark_Bailey

I recommend using the Fit Definitive Screening platform or the Generalized Regression (JMP Pro) instead of Fit Least Squares.

dbio27

Hi @Mark_Bailey, thanks for your help. I have played with the augment design tool a little before. I would like some more clarity on how to differentiate the vital few from the trivial many though when I use the augment design. When you first press on "Augment Design", a screen comes up that asks you to select your response and factor parameters. Let's say that only 4 out of the 10 variables come up as significant from the initial DSD experiment. Should I only select the 4 factors in this screen? Or should I select all 10, and then on the next screen when it asks for model terms I should start removing terms that aren't significant, and including interaction effects and polynomial terms that I am interested in. Or do these do the same thing?

Thank you

MathStatChem

Are the 10 factors you have chosen to experiment on all from the same process step. Typically, biopharma processes are multistep (cell lysis, clarification/TFF, refold, chromatographic purification, buffer exchange, ...). If the 10 factors are dispersed across several steps, then one nice advantage a DSD has is that these project into response surface designs for any 3 factors, and even if there are 4 factors you can build a nice prediction model. Also, in these process the factors from one step don't directly interact with the factors from another step. The can interact "indirectly", as the output from 1 step (which may be dependent on factors from that step) is an input to another step. This "indirect" interaction is often smaller than any process factor effect, however.

Let's say that step 1 has 3 factors, step 2 has 3 factors, and step 3 has 4 factors. Then you can run the DSD on the full process (Steps 1-3), characterize and measure the product quality at each step, and build many different models.

A. A model to predict product quality after Step 1

B. A model to predict product quality after Step 2. You can use the resulting quality measurements from Step 1 as covariates in this model.

C. A model to predict product quality after Step 3. You can use the resulting quality measurements from Step 1 or Step 2 as covariates in this model.

Don't forgot to consider blocking when doing this type of work. Often the block is associated with the analytical run used to measure the material coming out of the experiment, and you can block the experiments on "assay run" to account for any potential bias or confounding that will come from the assay.

All of that being said, I would definitely budget for follow on experiments, as often the DSD doesn't give you all the information you want about the process, or you may discover that you didn't choose the best ranges for experimentation, etc...

dbio27

Thank you @MathStatChem . Unfortunately no it's 10 factors for the same step of process (production with cell culture). I definitely don't believe all 10 will be important, but it could be somethingl like 4-6 (although mayb even less). We just aren't sure because we don't have a whole lot of data with this process.

Mark_Bailey

Be careful.

It is more than potential interactions among factors across more than one processing step that should be considered. It is likely that the early steps represent a restriction on the randomization of the runs that involve the factors in the later steps. If that is the case, then a DSD is not appropriate. A custom design with appropriately defined hard to change and very hard to change factors is likely to be more realistic and effective.

Improperly considering the multiple levels of randomization and ignoring the additional random effects in the model can compromise the inferential information from the analysis.

I apologize if I misunderstood your suggestion.

MathStatChem

That's a good point. If you are not going to carry material all the way through the steps, then for that number of factors, you might be better off to just characterize each step separately.

I've done it both ways.

MathStatChem

I'm guessing that you already have a good idea on what is important and what interactions may be present.

If you haven't already, it might be good to make an "interaction" map. Just go through each pair of factors and assess if you think they will interact. you can also consider if there will be an max or min output result in the middle of the design range. If that is true, then likely you will also have at least one quadratic effect. Then count up the main effects (10 in this case) and the number of potentially important interactions and quadratic effects, and make sure that number is less than the number of runs in the experiment. You can also check to see if the two factor interactions are strongly correlated with each other, and if so you might do some shuffling of the assignment of the factors to the DOE columns. Another advantage of adding the +4 additional runs in this case (10 continous factors) is that it creates a design with less confounding among the two-factor interactions and quadractic effects.

P_Bartell

I'm replying here after reading all of @Mark_Bailey and @MathStatChem 's input. I concur with everything they are advising or discussing. My recommendation was 'go slowly'. Start with a 'first' experiment...and I think we are all suggesting a DSD if that works logistically for you, as well as from an experimental design point of view. What I'm suggesting for after the DSD and it's results (statistical and practical) is...Pause. Take a deep breath with your team. Ask the following questions:

1. What have we learned wrt to solving the practical problem at hand?

2. Given what we've learned, what is the next logical step to solve the problem?

The answers to 1. can be multiple, conflicting, or just plain useless (but even then, one of the sage engineers I worked with when an experiment showed 'nothing' informed us all, "Hey, knowing an is 'not' can be just as valuable as knowing an 'is'." Try not to get too wrapped up in picking a specific DOE method (optimal design, classic RSM, more screening) at this point. Instead fit the problem with this hard won 'new' knowledge to the optimal next analytics steps. I think this is the essence of Mark's comments. DOE or otherwise. For example, what if you learn that your measurement system for the 'drug effect', that seems to be paramount here, is extremely noisy, making it hard to detect a signal above the noise? Subsequent DOE work could still be problematic...maybe it's resources better spent on the measurement system first? Or what if some factors and the levels you've chosen are persnickety? 'Persnickety' is a technical term I use when factors in an experiment are difficult to control, set, or have lots of known and unknown sources of variablity. In working with my biotech JMP process/product design customers over the years...any time they worked with 'cells'...well 'persnickety' was being polite.

dbio27

Thank you. I think I have a good idea of the path forward now, and will plan to use a DSD as the first run.