Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Hi,
For a JMP Inspire session with Danone, two users asked me to talk about using Space Filling Design in conjunction with Machine Learning algorithms. They asked me how to determine the optimal number of runs.
Would you have a strategy or an empirical approach for determining this?
Thanks for your help
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Not sure whether the application of our own Space Filling Design in our own XGBoost Add-in is relevant to your question.
If your "number of runs" is about the following two entries (circled) in the Add-in, then I guess that would be a kind of response surface problem. And what @malcolm_moore1 suggested is also related.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?+1 on previous replies and presuming this is related to hyperparameter tuning in machine learning, this could also be an opportunity to send feedback and ideas to @chris_gotwalt1 on Bayesian Optimization / Adaptive Learning.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?If we're moving into the stochastic world, the attached paper is likely relevant, and really still an open question. In particular, if you head to Section 4 on experimental design:
Suitable recommendations for the values of n1; n2;N and the replicates at each distinct input are lacking (in Ankenman et al. (2010) the recommendations are ad hoc) and, as for one-stage experiments, open for study. A third-stage design (or indeed, any multi-stage design) can be constructed by repeating stage 2 in the above process.
Often what I've seen is that you start with "enough" points in your initial design (based on the complexity, which could be a substantial amount), and then let Bayesian Optimization take over until you stop seeing an improvement - which you don't always know.
Certainly an open question in general though - how much exploration should we do with space-filling. In the 2nd paper I've attached, the finding was that in some cases practitioners are better off sticking to simpler and more classical models.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Hi @Steph_Georges, if the customer is interested in exploring how space filing designs can be used for the optimization of ML hyper parameters you may find my recording below valuable:
https://community.jmp.com/t5/JMP-in-the-Upstream/JMP-in-the-Upstream-7-Optimizing-your-Python-machine-learning/ba-p/467039
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Hi Stephane,
Space Filling Designs are usually associated with computer simulation experiments with a deterministic response (no random error or you get the same response value if you run identical factor settings) see this paper.
Perhaps you can find out more details about their experimental situation as it doesn't read like a typical space filling design problem and perhaps there is an alternative more efficient solution.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Good responses but we are all guessing at the user's intent. I could imagine different uses of Space Filling + Algorithm (training, tuning, testing, sensitivity testing, etc. etc.). The next question would be optimal number of runs for what? We often can't answer questions without a lot more information.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?@peng_liu @russ_wolfinger Can't help but go down the rabbit-hole, but this a particularly intriguing problem that I'm unaware of any literature (but please send my way if you do know of). For the inner folds, you want to design in a way that finds competing settings that may be chosen as a "winner" in one of the inner folds in subsequent designs. Otherwise you're wasting resources on hyperparameter settings that will have no impact on what is ensembled at the end.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?@Steph_Georges in a former life we used Space Filling designs as others described for deterministic experiments. We used either Gaussian Process or Neural Nets for the models. We preferred Neural Nets because of the flexibility of the models which were much better for working with the, typically, highly non-linear results we would encounter.
Gaussian Process is great for modeling if you are looking for the exact value of your data points, but it is not always great for modeling what is going on between data points. There is no noise in deterministic experiments, but there can be bias and this is where GP can be less effective than NN.
The optimal number of runs is a different animal. We always used the 10X the number of factors rule of thumb, but that is not to say it is optimal number, it was something that worked for our needs.
HTH
Bill
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Good evening,
Sorry, I realize that I wasn't very explicit and that my message is open to a lot of interpretation.
Danone (and other companies such as L'Oréal, for example) would like to use the methodology of Mixture DoE combined with new-generation algorithms as part of their formulation development.
The trend is well explained in this presentation by Victor Guiller from L'Oréal (at French User Group and Manchester Discovery Summit), who shows the use of Space Filling Design and the SVM algorithm to model behavior with an unknown a priori model (model-agnostic approach, prior knowledge of the model's shape is no longer necessary): https://community.jmp.com/t5/Online-Abstracts/Synergy-Between-Design-of-Experiments-and-Machine-Learning-for/ec-p/708147#M2
The use of Space Filling Design enables a regular paving of the area to be investigated ... which helps to better guide Machine Learning models (points too far apart leave too much freedom for interpolation, leading to aberrant response surfaces from a business expertise point of view).
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?I will second @peng_liu 's comment. One use of space filling designs is to optimize the tuning parameters so that you can get a better model fit.
If the question, however, is related to using a space filling design for an experiment, then I think the current best way to do that is to create space filling designs with varying number of design points and then use Compare Designs.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?I think you have to careful about the type of design you use with SVEM, especially for mixture optimization problems. You want to make sure you have good and evenly spaced coverage of the entire mixture space that you want to optimize over. And it will depend on the number of factors, including the expected error in the experimental process and measurement.
Evaluate Design can help with this, especially when you are comparing different designs.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Hi all,
I am getting similar requests from customers and the motivation comes from the proposed new way of analyzing Mixture DoE with Space Filling Designs with SVEM.
In Manchester, Andrew Fish from Johnson Matthey presented a use case of this: https://community.jmp.com/t5/Abstracts/A-Mixture-Process-Experimental-Design-and-SVEM-Analysis-for-an/ec-p/708143
This seems very appealing to others dealing with mixture problems. Johnson Matthey is working with the Predictum Addin.
What I recall from their materials is that generally the approach of using Space Filling Designs with Neural requires much less runs compared to Scheffé Polynomials. But I would be very happy to learn more about the set up of Space Filling Designs for Mixture Problems, like appropriate number of runs.
Best
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?I spoke with a JMP user who had said that Predictum told him that 3 runs per factor is adequate using their Space-filling/SVEM/Neural Nets approach. Maybe he misunderstood. That seems unrealistic to me, although it might be okay in some situations.
My view would be that if you are choosing to use a model-agnostic approach to designing experiments then you need to be okay with not knowing how many runs you will need. And that is fine as long as you view it is a sequential approach.
With model-based DOE you can estimate the power of your experiment, as long as you have good estimates of signal and noise (we often don't). With model-agnostic design you don't know what effects you are trying to estimate (by definition), so power calculations are meaningless.
I suspect that the best you can do is to build experience of how many runs per factor are typically required for the kind of systems that you are experimenting on.
Re: Space Filling Design + ML algorithms - How to estimate the adequate number of runs?Following up on great ideas and references from @Ryan_Lekivetz , one approach that seems sensible for evaluating a design + ML model combination is repeated K-fold CV with K set to a high number to only leave out 2 or 3 design points for each fold. This should effectively and robustly test the performance of both the design and model simultaneously and rapidly find targeted regions of weakness or even breakdown.
I've attached a copy of Andrew Fish's nice DSE example with 26 runs, implementing repeated13-fold CV / leave-2 out with embedded scripts for both XGBoost and Torch.
Alas, I don't think Model Screening works properly in this case: 1. Straight K-Fold CV is leaky 2. Nested K-fold breaks down when K or L are set too high (I've investigated the code and have a fix in my playpen but have not pushed it). @John_Sall is aware of these issues and it would not hurt for him to hear about them from someone other than me 
Hopefully we can improve things in 19.
