nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Space Filling Design + ML algorithms - How to estimate the adequate number of runs?

Steph_Georges

Hi,
For a JMP Inspire session with Danone, two users asked me to talk about using Space Filling Design in conjunction with Machine Learning algorithms. They asked me how to determine the optimal number of runs.

Would you have a strategy or an empirical approach for determining this?

Thanks for your help

Accepted answers

All comments

malcolm_moore1

Hi Stephane,

Space Filling Designs are usually associated with computer simulation experiments with a deterministic response (no random error or you get the same response value if you run identical factor settings) see this paper.

Perhaps you can find out more details about their experimental situation as it doesn't read like a typical space filling design problem and perhaps there is an alternative more efficient solution.

peng_liu

Not sure whether the application of our own Space Filling Design in our own XGBoost Add-in is relevant to your question.

If your "number of runs" is about the following two entries (circled) in the Add-in, then I guess that would be a kind of response surface problem. And what @malcolm_moore1 suggested is also related.

russ_wolfinger

+1 on previous replies and presuming this is related to hyperparameter tuning in machine learning, this could also be an opportunity to send feedback and ideas to @chris_gotwalt1 on Bayesian Optimization / Adaptive Learning.

SamGardner

I will second @peng_liu 's comment. One use of space filling designs is to optimize the tuning parameters so that you can get a better model fit.

If the question, however, is related to using a space filling design for an experiment, then I think the current best way to do that is to create space filling designs with varying number of design points and then use Compare Designs.

karencopeland

Good responses but we are all guessing at the user's intent. I could imagine different uses of Space Filling + Algorithm (training, tuning, testing, sensitivity testing, etc. etc.). The next question would be optimal number of runs for what? We often can't answer questions without a lot more information.

Ryan_Lekivetz

If we're moving into the stochastic world, the attached paper is likely relevant, and really still an open question. In particular, if you head to Section 4 on experimental design:

Suitable recommendations for the values of n1; n2;N and the replicates at each distinct input are lacking (in Ankenman et al. (2010) the recommendations are ad hoc) and, as for one-stage experiments, open for study. A third-stage design (or indeed, any multi-stage design) can be constructed by repeating stage 2 in the above process.

Often what I've seen is that you start with "enough" points in your initial design (based on the complexity, which could be a substantial amount), and then let Bayesian Optimization take over until you stop seeing an improvement - which you don't always know.

Certainly an open question in general though - how much exploration should we do with space-filling. In the 2nd paper I've attached, the finding was that in some cases practitioners are better off sticking to simpler and more classical models.

analyzing stochastic.pdf

Quality Reliability Eng - 2022 - Sunder - Robust sequential experimental strategy for black‐box optimization with.pdf

Ryan_Lekivetz

@peng_liu @russ_wolfinger Can't help but go down the rabbit-hole, but this a particularly intriguing problem that I'm unaware of any literature (but please send my way if you do know of). For the inner folds, you want to design in a way that finds competing settings that may be chosen as a "winner" in one of the inner folds in subsequent designs. Otherwise you're wasting resources on hyperparameter settings that will have no impact on what is ensembled at the end.

Bill_Worley

@Steph_Georges in a former life we used Space Filling designs as others described for deterministic experiments. We used either Gaussian Process or Neural Nets for the models. We preferred Neural Nets because of the flexibility of the models which were much better for working with the, typically, highly non-linear results we would encounter.

Gaussian Process is great for modeling if you are looking for the exact value of your data points, but it is not always great for modeling what is going on between data points. There is no noise in deterministic experiments, but there can be bias and this is where GP can be less effective than NN.

The optimal number of runs is a different animal. We always used the 10X the number of factors rule of thumb, but that is not to say it is optimal number, it was something that worked for our needs.

HTH

Bill

nick_shelton

Hi @Steph_Georges, if the customer is interested in exploring how space filing designs can be used for the optimization of ML hyper parameters you may find my recording below valuable:

https://community.jmp.com/t5/JMP-in-the-Upstream/JMP-in-the-Upstream-7-Optimizing-your-Python-machine-learning/ba-p/467039

Steph_Georges

Good evening,

Sorry, I realize that I wasn't very explicit and that my message is open to a lot of interpretation.

Danone (and other companies such as L'Oréal, for example) would like to use the methodology of Mixture DoE combined with new-generation algorithms as part of their formulation development.

The trend is well explained in this presentation by Victor Guiller from L'Oréal (at French User Group and Manchester Discovery Summit), who shows the use of Space Filling Design and the SVM algorithm to model behavior with an unknown a priori model (model-agnostic approach, prior knowledge of the model's shape is no longer necessary): https://community.jmp.com/t5/Online-Abstracts/Synergy-Between-Design-of-Experiments-and-Machine-Learning-for/ec-p/708147#M2

The use of Space Filling Design enables a regular paving of the area to be investigated ... which helps to better guide Machine Learning models (points too far apart leave too much freedom for interpolation, leading to aberrant response surfaces from a business expertise point of view).

SamGardner

I think you have to careful about the type of design you use with SVEM, especially for mixture optimization problems. You want to make sure you have good and evenly spaced coverage of the entire mixture space that you want to optimize over. And it will depend on the number of factors, including the expected error in the experimental process and measurement.

Evaluate Design can help with this, especially when you are comparing different designs.

Florian_Vogt

Hi all,

I am getting similar requests from customers and the motivation comes from the proposed new way of analyzing Mixture DoE with Space Filling Designs with SVEM.

In Manchester, Andrew Fish from Johnson Matthey presented a use case of this: https://community.jmp.com/t5/Abstracts/A-Mixture-Process-Experimental-Design-and-SVEM-Analysis-for-an/ec-p/708143

This seems very appealing to others dealing with mixture problems. Johnson Matthey is working with the Predictum Addin.

What I recall from their materials is that generally the approach of using Space Filling Designs with Neural requires much less runs compared to Scheffé Polynomials. But I would be very happy to learn more about the set up of Space Filling Designs for Mixture Problems, like appropriate number of runs.

Best

Phil_Kay

I spoke with a JMP user who had said that Predictum told him that 3 runs per factor is adequate using their Space-filling/SVEM/Neural Nets approach. Maybe he misunderstood. That seems unrealistic to me, although it might be okay in some situations.

My view would be that if you are choosing to use a model-agnostic approach to designing experiments then you need to be okay with not knowing how many runs you will need. And that is fine as long as you view it is a sequential approach.

With model-based DOE you can estimate the power of your experiment, as long as you have good estimates of signal and noise (we often don't). With model-agnostic design you don't know what effects you are trying to estimate (by definition), so power calculations are meaningless.

I suspect that the best you can do is to build experience of how many runs per factor are typically required for the kind of systems that you are experimenting on.

russ_wolfinger

Following up on great ideas and references from @Ryan_Lekivetz , one approach that seems sensible for evaluating a design + ML model combination is repeated K-fold CV with K set to a high number to only leave out 2 or 3 design points for each fold. This should effectively and robustly test the performance of both the design and model simultaneously and rapidly find targeted regions of weakness or even breakdown.

I've attached a copy of Andrew Fish's nice DSE example with 26 runs, implementing repeated13-fold CV / leave-2 out with embedded scripts for both XGBoost and Torch.

Alas, I don't think Model Screening works properly in this case: 1. Straight K-Fold CV is leaky 2. Nested K-fold breaks down when K or L are set too high (I've investigated the code and have a fix in my playpen but have not pushed it). @John_Sall is aware of these issues and it would not hurt for him to hear about them from someone other than me Hopefully we can improve things in 19.

DoE_SpaceFilling_Results.jmp

chris_gotwalt1

I am piping in to summarize what we found while I was collaborating with UNH on SVEM. We did a lot of simulations comparing sample sizes, designs, and estimation methods. These were all continuous factor RSM situations, without mixtures, but since then consultants and JMP customers have seemed to zero in on mixture designs with SVEM and are having a lot of success with it.

We compared SVEM+FS ( forward selection SVEM), SVEM+Lasso, with standard single-shot Forward Selection (FS) and the Lasso. Early simulations also looked at the Fit DSD approach on definitive screening designs. Our comparisons were always based on independent test set accuracy, and we did not consider things like parameter estimation or inference.

Overall SVEM+FS performed the best whenever the design was supersaturated in the interactions or other higher order effects like quadratic terms or partial cubics ( think x1 * x2^2 ). This is in the sense that these situations SVEM+FS usually did the best overall, and when it didn't there wasn't a large difference. When there are enough runs to estimate the full model (i.e. a central composite design ) SVEM+FS and Forward Selection performed similarly. So, if you can fit the full model, I would usually go with FS, but we didn’t find any situations where something clearly beat SVEM. So, no reason not to try both.

Early on we looked at a lot of variations of SVEM before finding the model averaging approach performed the best. We did different kinds of pruning the final model. One surprise was that if you have added a random null factor that is independent of Y it was better, as measured by test set RMSE, to leave the null factor in the final model, than to remove it and all terms that appeared less often than the null factor. At that point we dropped the null factor concept and moved forward with straightforward model averaging. This, like a lot of the things we didn’t investigate further was because we had a finite amount of time and resources, it all having to be in the scope of a dissertation. The simulations were very time consuming since they were a simulation of a simulation based method.

SVEM is explicitly *not* a screening approach. In informal simulations we didn't write up, we did not see an advantage to it with 2-level designs like fractional factorials. But, we didn't look far in that direction since we were focused on RSM type prediction problems.

We evaluated the Fit DSD approach, and we didn't find it to do well as a one shot predictive modeling approach relative to the other approaches as measured by test set accuracy. That said, DSD's are dual purpose designs for both identifying active factors with the ability to fit some RSM type models with very few runs. We didn't evaluate its ability to identify the active set, as there are other papers out there that have already done that.

When the model is supersaturated for the data you have, the design that performed the best amongst the alternatives we considered was Bayesian I Optimal where the main effects were 'Necessary' and the higher order terms were "If Possible". Latin Hypercube designs also did surprisingly well, and Bayesian D-Optimal designs did rather poorly in some situations, with negative median RSq's.

We looked at different percentages of active effects, including situations where the design was supersaturated not only in potential factors, but supersaturated in active factors. We also looked at S/N Ratios of parameters to measurement error, and generally speaking, SVEM+FS and a Bayesian I-optimal design generally either was the best or was equivalent to the best.

SVEM isn't magic or voodoo. There were situations where nothing we considered did well. We actively sought to find situations where it didn’t work so that we would get a sense of where that happened, what it looked like when it did go wrong, and see if any other methods did better somehow. As you would expect all methods did poorly when the sample size was small relative to potential effects, and there were a lot of active effects, and the measurement error was large (that is, three times the variability in our simulated regression parameters). I don't think anything will do well in those situations. The relationship between minimum sample size needed, and things like %active effects, and measurement error is complicated and to my knowledge analytically intractable. Customers using any approach to modeling with highly supersaturated designs will need to have at least an initial risk tolerance, and like Phil Kay said, the minimal requirements for these designs is context dependent and up to the customer to work out.

russ_wolfinger

Thanks much @chris_gotwalt1 for the great summary and insights. To tie our replies together, would love to see one or more SVEM applications surfaced in Model Screening. I quickly tried to hack the code to include one of the GenReg ones but it bombed--there must be more complex bookkeeping involved.