How are you planning on showcasing Bayesian Optimisation?Hi folks,
Curious to see how people are planning to pitch and position Bayesian Optimisation in their SE workflow, are you going to include it in DoE sessions? Link it into production improvement? Show it everywhere/anywhere? Where do you see yourself talking about it?
Will be interesting to see where people think it will 'fit in' to the way they currently work/talk with customers and prospects, or if a new way will come about!
Thanks!
Ben
Re: How are you planning on showcasing Bayesian Optimisation?Very good question, @Ben_BarrIngh .
It would be interesting to hear what we have learned from talking to customers about it so far.
I've talked to some customers that are already interested in the method. I've also talked to some of our bigger customers where BayesOpt can provide additional value and/or be a reason for more Pro. Positioning BayesOpt in both of these situations is relatively simple. And those should be our priorities.
I have not talked about BayesOpt with any of our very large number of small customers (remember our median number of JMP users at an account is somewhere between 2 and 3) or with any prospective accounts. How do we talk about BayesOpt in those situations? Should we even talk about BayesOpt with new prospects?
@Ben_BarrIngh - can you give a typical example of the kind of accounts that you are thinking about?
Re: How are you planning on showcasing Bayesian Optimisation?Hi @Phil_Kay ,
Thanks for the reply - the areas I'm particularly interested in how we start BayesOpt conversations are:
A) New customers who want to simply improve their experiments - do we offer out BayesOpt in the same breath as DoE? Do we focus on the end goal (understanding vs. reaching an optimum) and then distinguish? What value statements can we make (metric based, time saved, resource % reduced) that help make a convincing argument?
(On that note, @malcolm_moore1 found this infographic https://www.active-learning.uk/2024).
Current customers who know DoE - For customers who know DoE, but aren't Pro customers - what pains do we look to target that brings in BayesOptimisation to the conversation. If we're running a DoE inspire - do we talk about the DoE tools and then jump into Bayesian Optimisation? Or does that confuse things? Do we rename our inspires to 'Modern Experimentation' or similar to get the framing right?
Thanks,
Ben
Re: How are you planning on showcasing Bayesian Optimisation?Hi guys,
from what I have heard from customers so far, not all find sequential experimentation feasible. Some of them have a time constraint and prefer testing in parallel, even at the expense of a couple of more runs.
I would talk to customers about Bayes Opt if I knew that either their runs are very expensive/done with raw materials that they do not have enough of, or if the underling model is very complex. If time is a concern and they can not do their runs in parallel (e.g. due to a lack of equipment), then I would also talk to them about Bayes Opt.
Many come to us with DoE in mind. If the criteria above are met, I would include Bayes Opt in a DoE demo. As discussed in Amsterdam, we don't want to make it look like a "better alternative" to DoE, but rather a method that is more suitable in some cases, where in others DoE would be the method of choice.
Re: How are you planning on showcasing Bayesian Optimisation?Hi,
I'd like to add these Situation Summaries that Florian, Stéphane and Owen developed for BayesOpt to the conversation. They've also added an elevator pitch after the summaries that list some of the challenges customers may face for which BayesOpt could be a solution:
-
High experimental cost limit number of possible experimentation
-
Expensive or little available raw material doesn’t allow endless experimentation
-
Time Pressure doesn’t allow for extensive studies
-
Optimization in running production doesn’t allow failure
-
Complexity and high dimensionality require systematic approach
Re: How are you planning on showcasing Bayesian Optimisation?Thanks for starting this discussion @Ben_BarrIngh and great to see what we have learned in such a short time. Big similarity between what @Lisa_Daroque and @HadleyMyers shared regarding target problems for where BayesOpt might be appropriate. At a company level both tools (DoE and BayesOpt) are needed -- lets encourage use of the appropriate tool for the specific problem.
Please keep sharing learning experiences with BayesOpt and JMP 19 generally.
Re: How are you planning on showcasing Bayesian Optimisation?Hi @Lisa_Daroque ,
Great graphic, that really helps flesh it out. For those with scarcity/resource constraints do you think that we would need to 'suspend' belief with BayesOpt? I wonder if people would see it as 'magic' when we claim it will reach optimums with a small/smaller number of experiments.
Cheers,
Ben
Re: How are you planning on showcasing Bayesian Optimisation?Great topic. I would love to hear more about this as JMP 19 is released next week. I like @Ben_BarrIngh idea of showing Bayesian Optimization in conjunction with DOE. @bradleybowen has workshop materials he will share that have three case studies of varying complexity. He will need to make sure everything works as expected in the production release, but you should have access shortly after the release.
Re: How are you planning on showcasing Bayesian Optimisation?Thanks @Ben_BarrIngh all the other for your feedback and contribution.
I went through all the 6 videos in the page you posted and I remained with a bittersweet feeling;
Five of six come from our customers (if I’m correct):
- All of them have a strong interest for BO (that’s good)
- All of them are already using heavily open source for BO (that’s bad)
I noticed, as you already said, that they compare BO with DOE (see a screenshot below), so I want to add 2 more questions to Ben’s:
- Do we see the risk to address people to BO with a solution already available in open source – already used in their company?
- What we can do to avoid it and justify an extra spent for JMP Pro to use BO?
Thx
Marco
Re: How are you planning on showcasing Bayesian Optimisation?Thanks for digging into this @MarcoSalfi . Here are my thoughts regarding your two questions:
- Do we see the risk to address people to BO with a solution already available in open source – already used in their company?
- What we can do to avoid it and justify an extra spend for JMP Pro to use BO?
I see nothing different to the usual sales situation here where there is often an incumbent solution of some sort, even if it is Excel or pen and paper. If possible try and get an understanding of their pains and what it is they can't fix/accomplish/avoid with current solutions. If they don't have a pain or they are resistant to change then move on. If they do have a pain that we can align to with BayesOpt (or other capability) then try to complete a specific situation summary for their circumstances. In the case of BayesOpt the consequence of not changing will need to be big enough to motivate for paying the license fee for Pro and a situation summary will be most helpful in advancing such opportunities.
It will be interesting to get different/additional views regarding your questions.
Re: How are you planning on showcasing Bayesian Optimisation?My two cents on open source solutions for BayesOpt (or data analysis in general) is that maintaining code that relies on open source libraries is a nightmare.
Especially if you're using libraries developed in academic environments and suddenly there's no investment in the library anymore.
Besides, you need a mix of many libraries, because many of them only offer one part of the solution.
From what I've seen in Python and R for Bayesian optimization, JMP's BayesOpt is a far superior solution.
Re: How are you planning on showcasing Bayesian Optimisation?@LuizaFabreti I wanted to add that there are other software options out there, namely Citrene, which offer an optimization approach that is based on Bayesian Optimization. We have a customer who was using this software but never got any meaningful results. They told us they had spent over $100,000, and probably a lot more in reality, to come up empty handed. They have since become beta users of the new Bayes Opt platform and helped drive a few of the innovations Chris and team have since implemented.
So, the moral of the story is that there are probably more folks out there trying sequential optimization than we think, and it never hurts to let them know that JMP now has a built-in Bayes Opt platform in JMP Pro.
Re: How are you planning on showcasing Bayesian Optimisation?Love to see this discussion going on. Have to throw a few comments in of course.
Competitors: There are a few commercial tools and also quite a few open source solutions. Important to note is that not all have the same scope and mostly the scope is pretty narrow or specific. BoTorch is what many open source solutions are built on.
The advantages I heard from customers so far about open source implementations:
- it's free (haha)
- it's more flexible, tailored to their needs
- it's easier to be bound into "Closed Loop Optimization tasks", where an experiment is suggested, executed, analyzed and another is suggested until a target is met
One of the solutions several our customers developed (credit to @JessicaHorsch for finding it) is BoFire (https://arxiv.org/pdf/2408.05040). The main author happens to be my customer and is obviously very convinced that this solution will be the only way chemical companies will experiment in the future, but equally acknowledged that black box optimization does not meet all the needs of scientists and engineers. And my champion at Evonik confirmed that he saw this lead to success but also to absolute failure (because it's blackbox).
Another situation was a customer asking for a way to implement historic data, or their current knowledge and even mechanistic information from reactions etc. to suggest new experiments.
To me, it now almost feels like BO is somewhere between the "oh this, yes we do it as well" and the absolute killer feature we need to show around whenever we can.
I think we need to carefully qualify first and get a sense of type of people we speak to. At the same time we can confidently advertise it, because it is a unique and powerful implementation and as always, combined with all JMP (Pro) can do it's nicely embedded in a powerful toolset.
