DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?Hi,
Hoping you might have some ideas for this interesting problem.
We have an objective to find the settings that optimise yield for a chemical reaction.
There are 5 categorical factors for different choices of reactants, reagents, ligands, solvents (factors table attached). The factors have between 3 and 12 levels.
There are 4608 possible combinations of the factor levels. It is not feasible to run all of them!
It is possible to carry out 48 runs simultaneously, taking about 2 days of lab work.
So, in the first stage we run a custom main effects screening design with 48 runs (attached).
This first block of runs uncovers some useful insight:
- Reactant_1 and Reactant_2 have the biggest effect.
- Some combinations give yields of >90%.
Distribution of %Yield and 5 Factors for 48 run screening experiment.Distribution of %Yield and 5 Factors for 48 run screening experiment.Distribution of %Yield and 5 Factors for 48 run screening experiment.
The question is: what next? What should the next 48 runs be?
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?Thanks, @Ryan_Lekivetz . I will definitely be in touch as I think this could make an interesting paper but we have only just begun to think about this.
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?@benfrancis asked about the idea of augmenting with the predicted model, so I figured I'd post it here in case it's of use to anyone else.
In the Augment_example.jmp, I created a simple 12-run design with 3 continuous, and 1 3-level factor. The simulated response was based on a coefficient of 3 for X3 and a 2 for the first level of X4 (so -2 for the third level). I saved the Predicted formula from Fit Model based on this original data set, and "converted" it into a disallowed combination by removing the colons and replacing the match with a reference to each level. I then set it to disallow point where the predicted response is less than 3. That is, the formula:
(-0.488118773193337) + 0.0500127592094168 * :X1 + 0.090943953251116 * :X2
+3.44923140690808 * :X3 + Match( :X4,
"L1", 1.81325700252147,
"L2", 0.117965718890636,
"L3", -1.93122272141211,
.
)
becomes:
(-0.488118773193337) + 0.0500127592094168 * X1 + 0.090943953251116 * X2
+3.44923140690808 * X3 + (X4 == 1) * 1.81325700252147 + (X4 == 2) *
0.117965718890636 + (X4 == 3) * -1.93122272141211 < 3
The "Augmented Design with disallowed.jmp" shows the resulting design and a double check that the predicted response values are all above 3. Note that it doesn't end up picking X4 at "L3" or X3 at -1. I have to see if there's a reason it's still not varying X3 slightly, but it gives an idea.
When it's mainly continuous inputs, I've also used this same approach with FFF designs to restrict the design space to where the model looks interesting.
Hope this helps,
Ryan
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?Hey @Phil_Kay ,
This is an interesting problem... I'm not up on the literature, but it seems that it might be worth publishing something based on what you're doing as a "what if" you treated it as a designed experiment. I've been thinking of avenues to talk about this type of issue for awhile (full factorial vs. a subset), so if you're not already down the publishing path, maybe we should chat. It reminds me of this talk from a few years ago:
http://www.acadiau.ca/~hchipman/talks/ChipmanSSC2015.pdf
Regarding the augmentation problem, please feel free to add something to the internal wishlist - it would be nice to remove levels in future experiments.
I've also had success in the past where I use the predicted model from the first design as a disallowed combination (which is a bit more of a pain with categorical, since you have to convert it). Where you take the predicted formula and disallow combinations where the predicted value is below some value (say 80). And as @nick_shelton mentioned, you can also disallow those levels as well.
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?By the way, the example comes from a published paper where they actually ran the full factorial (4608 runs!).
This is a really nice dataset (attached JMP table). Although it is difficult to know how accurate or precise the data are.
If nothing else you can make a nice heatmap:
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?Thanks. That is really helpful, @nick_shelton .
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?In augment design, you can't remove categorical levels but you can set a factor constraint to prevent new runs from using certain levels.
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?I like that idea, @benfrancis .
Ideally we would select a set of 48 runs that are:
- Predicted from our current model to have high yield
- Optimal for testing the model of interest
So maybe we could:
- Generate a very large candidate set of 1000s of runs (say, the full factorial) and then
- Select from this a subset of 100s of runs that are predicted to have yield above some threshold.
- Use these as a candidate set and Custom Design with covariate factors to determine the optimal 48 to test the model
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?Thanks @benfrancis and @nick_shelton .
Both good ideas.
Some confirmatory runs around the current predicted best settings would seem like a good idea. But I think we might wish to do more with the next 48 runs that we have.
Expanding on what Ben suggested I looked at the Scaled Estimates and there are category levels from some factors that are significantly worse than the others, e.g. Ligand[Xantphos].
(Xantphos is just the worst ligand ever, isn't it. Personally I would never use Xantphos for anything 
).
It would seem rational and justified to remove these category levels from further runs.
Similarly we would be justified in getting rid of 2 of the levels of Reactant 2 and just using Boronic acid.
Has anyone tried using Augment on categorical variables? I can't seem to figure out how to remove a category level before augmenting.
Cheers,
Phil
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?Good point @Nick, perhaps we could use the simulator to find out a set of predicted high yield (Obj>0.95) candidates? This leads to 23 candidate sets in the attached.
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?If you include the interaction between Reactant 1 & 2 into the model there's a combination of factors in the profiler which bring the yield to 99%. Instead of exploring other combinations of factors with subsequent DOEs perhaps it's best to now move into a set of confirmatory experiments to validate that those identified settings will yield 99% in practice.
Re: DOE with 5 multi-level categoricals. Best sequential approach to finding optimum?Hi Phil,
Thanks for highlighting this example, this seems important for a number of customers who focus on the big experiment rather than a sequential approach. And this is a head scratcher so am eager to explore what everyone thinks!
Considering whether we have grounds to screen out levels of factors could be useful. Using Effect Details (better than my Fit Y by X initial suggestion) and a T-K HSD (being as conservative as possible), maybe there's grounds to remove 6-Cl-Q (Reactant 1) and Trifluoroborate (Reactant 2) as levels? Also interesting is that whilst "None" for ligand is 2nd to lowest average yield, "None" for Reagent 1 is mid yielding. It's not the same as a null factor but perhaps focus towards if some reagent 1's that are negatively impacting could be useful in the next 48?
Kind Regards,
Ben
