Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergHere I describe my learnings from the Heidelberg Workshop with Chris Gotwalt on Bayesian Optimization:
Some short facts:
- 48 registrants - about 40 in the room
- Topic Bayesian Optimization
- lots of interest
- about 60% had JMP Pro
- Most have been familiar with DoE and targeted to optimize processes faster and with lest costs
Assumptions and Confirmations on Exotic Situations:
- Multiple responses, 𝑦_𝑖 (𝑥)
- All hands went up asking to confirm that is important for them and they need that
- “Batch” augmentation of more than one run
- Almost all have situations where they need to have the need for batch experimentation, means not just the next but e.g. the next 4, 6, or whatever number. Reasons for this is most of the time
- the experiment last too long to wait for the next suggestion,
- but also lack of material (not always available) is another reason.
- Expectations mentioned:
- how to shorten time in development
- Analytical department (Design Expert user) goal
- to persuade colleagues to use DOE easier as they say so many runs cannot be done.
- Give users options for sth else than DOE, why should I do experiments that bring me nothing I just want a good result
- Cost -> less runs
- What do you hope to get?
Questions from the audience:
- What if factors are not important?
- Answer: It will ignore factors that do not have an impact. Variable importance option will help to show the power ordered factors
Can responses be weighted? (another "exotic" case)
- Answer: Yes, it will be applied in the bayes opt desirability in the profiler. Competing goals or complicated sparsity can be handled, as well match target/spec limit goal
How will be local min/max issues be handled?
- Answer: Compare it a bit with Newton, this might happen, but if you are not happy, you can easily explore other regions e.g. with high variance.
Can I use categorical responses/factors? (another "exotic" case)
- Answer: Yes, but correlation won't be based on distance as for continuous responses
What if I know the response will be very noisy? How to handle this here? (another "exotic" case)
- Answer: You need to develop strategies: Do a MSA first to assess the variance and use replicates to average out the variability
Can I add new factors? (another "exotic" case)
- Answer yes, even in the middle of the project
Use cases:
- Chris G.: Formulations and mixtures is the number one use case as all the confounding and constraints are not an issue anymore. Because it is distance and not design space dependent! Inflation factors don't matter at all.
- Customers mentioned:
- PID controller tuning: One run takes 15-30min, but it is cumbersome to go in and out every time, so batch runs are very important to them (compared to basic bayes opt solutions with just the next run)
- Hyper parameter training and modeling
- GMP environment where you cannot vary too much and are limited e.g. to 2 batches to be run at all
- Can have huge effect in Research (Pharma), may less in e.g. Process Characterization
- Persuade colleagues to use DOE, as it is easier and you can overcome they as they say so many runs cannot be done.
Suggestions: @chris_gotwalt1 @DanielValente
- P&G/BASF: Would be great to have the option to add prior information (e.g. from known chemical reactions/formulas or that you know it is for sure a linear model) to influence desirability curve
General summary:
- Big interest
- Our platform seems to fit well into the landscape between Bofire (for Python programmers) and self-developed solutions (made very easy for business people)
- It is much more than most Bayesian optimization tools can do - talk about the "exotic" Bayes Opt situations
Further comments:
- How to handle questions about the underlying selection and math?
- People want that and see that it helps lots of JMP users and want to have this in JMP. That is a big reason to talk about the value they could get if all those users would use that and do a ROI calculation to see how a ELA/certain number of Pro licenses compare. E.g. let the advocate show the difference from past experiments to what if we would have done BO after the first experimentation. Did it lead to some other route/find the answer earlier/when was the point where we went into the wrong direction? It is about convincing others (same engineers who doing the experimentation over and over again with their assumptions.
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergOne more comment: Organizations will be probably have already some kind of solution in use, e.g. BoFire (python package) or sth. simpler. Most of them do not have all of the exotic situations in their tools or are for programmers.
We are there to take a good bunch of the market with what we have now.
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergGreat sum up of the feedback. Just some more Q&A-s I ve noted.
What if the table comes from somewhere else?
- Answer: No worries you can add all constraints in the profiler
An example to show how the profiler evolves through runs ?
- Answer: Yes it would be nice! Chris said they will maybe add this in the future. In his presentation he managed to show this evolvement of the profiler manually by using multiple slides but it was a lot of work.
Do you give weights as well when you optimize?
- Answer: Yes just like we do in the profiler, you can change them on the fly, and if you change the desirability ( for example maximize in the abrasion case) then the desirabilities for bayes opt will automatically adjust.
What if the ranges are not good enough or narrow?
- Answer: You can just change the ranges in the profiler and then it accounts for them
Can we save the formula/prediction profiler?
- Answer: There is no direct way to do that. One must go back to the data table and Open the Gaussian Process platform where there you will be able to save the model (bare in mind that this will be a different formula/model that the one created from the Bayesian Optimization platform)
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergThanks for sharing Martin - Any thoughts on the flow of the workshop itself? Did you show an example then have the audience work through an example with you? Do you think we should add or change one of the examples to do a batch of 4 runs at once instead of a single run?
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergThank you all for sharing your insights from your discussions around BayesOpt. I’m sharing a tension stated by some of attendees of the London workshop: We have many scientists wishing to give Bayesian Optimisation a try, but I’m concerned they will do something wrong in the application of the method. This made me wonder if some of our advanced users are statistical gatekeepers and may discourage use by mainstream users.
Perfection is a noble and hard to achieve goal. Helping scientists and engineers to ditch trial and error experimentation is the worthy outcome we might be after. To help scientists and engineers achieve more fulfilling work and deliver more predictable and often better business results (than trial and error approaches). Might it be possible to encourage our statistical gatekeepers to focus less on perfection in the application of statistical experimentation methods and more on supporting scientists and engineers in their evolution of change and in a way that works for the many. As part of the evolution scientists and engineers may discover they need both DoE and BayesOpt, but the starting point should be down to them and their manager.
Please look out for the statistical gatekeeper. Try to help them think about the challenge of helping scientists and engineers to evolve their experimental approaches in a way that has most appeal to the scientist or engineer. If the statistical gatekeeper doesn’t wish to evolve then perhaps we need to find other buying influencers who do have an urgent need to change from trial and error approaches. Find a way of gaining momentum, while there is a buzz around Bayesian Optimisation in the scientific community.
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in Heidelberg@malcolm_moore1 - It's interesting that you bring that up. A couple of my accounts have these types of people in them. A few of us on the east coast have been positioning BayesOpt as a tool that's best used by the SME's running the experiments - with the SME's themselves - rather than pitching it as a technique for the Stats teams. The thought is that it is suggesting possible new experiments to help the SME's navigate the multidimensional nature of manufacturing data. It also seems to do really well with something like a DSD as a seed set - suggesting a hybrid DOE strategy that might sit well with manufacturing teams. Since, it's using (mostly) automated techniques in JMP.
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergMakes great sense to me @MikeD_Anderson, to focus on the SME's when we get resistance from the statistical gatekeeper who are more likely to be in the larger enterprises. Please let us know how things go pitching BayesOpt to the SME's.
Perhaps not give up on the larger enterprises before seeking to find the buying influencers who do need BayesOpt and are happy with an incremental improvement compared to their day in a life before BayesOpt.
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergHi Pete,
The flow of the workflow was Chris about Bayes Opt in general, First example, lunch, Chris mor detailed , second example hands on, third example brief, wrap-up.
I think the first example (simple one) is a good entry point for most of them. And the second case they work on is more hands-on, from the feedback it was good for the majority. Don't think it is really necessary to adapt the use case, as long as one mentions the option. Chris showed that part in some slides as well, so it is covered.
However, for our SE demos/inspires it would be good to have the batch use case as a comprehensve example ready to show.
That said, I still believe we should be showing just a very brief pitch in the platform and talk /discuss more on customer's potential use cases to apply these. Mainly in the direction it has been mentioned already by Malcolm and Mike
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in Heidelberg@malcolm_moore1 - I thought that was interesting too. I suspect that these gatekeepers are also concerned that their DOE expertise might be made redundant. What I heard was that people are already trying Bayesian Optimization but they are doing so using a variety of open source, which is very concerning for the gatekeeper. Assuming that we have built trust and credibility with the gatekeeper, then enabling experimenters with BayesOpt in JMP Pro should be a more appealing solution.
What would you prefer:
- guide your colleagues to explore the possibilities of this exciting new approach in JMP Pro?
- let your colleagues mess around with Bayesian Optimization on their own in Python?
I agree that in other cases, finding ways to get around the gatekeeper will be the right thing to do.
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergGreat to see BayesOpt taking off! If anyone is interested in an easy way to demo it for hyperparameter optimization:
1. Run XGBoost with Autotune checked.
2. In the bottom of the output is a table containing stats on each run. Right click > Make into Data Table.
3. Run BayesOpt on this table to determine next best candidate hyperparameters.
4. Load these back into XGBoost manually or use the Design Table option under Advanced.
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergThanks @malcolm_moore1. This is something I have come across a few times in the past weeks. I found two reasons for this (both with the same "gate keeping" result):
1. Fear of loosing control/power (because it's their expertise currently etc.)
2. Fear people will do random experimentation instead of DOE (they have been preaching for people to do DOE instead of OFAT etc.)
These were two different persona, but both types were more or less against broader use of BO.
In both cases I had users saying "oh actually that (iterative, small batches of experiments) is exactly how we operate and do our experiments - we should be using BO".
For the second group of "gate keepers", we can make them more comfortable by letting them know that BO is actually a low entry barrier way to bring people to data driven decision making. Because if they don't know much, using BO they essentially do DOE (without knowing). Demo BO with a factors table as starting point (2-3 runs). So, it actually helps them on their mission to make more people use DOE or what I would maybe call smart experimentation methods.
For the first group, yeah probably best find someone else to speak to 
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergGreat insight Florian. For Q4 I suggest we focus on the potential users of BayesOpt who are motivated and have urgency to change. Close these opportunities then support them in demonstrating success.
When we have more time (next year) perhaps see if we can explore the personal wins for the "gate keepers". I hear about reduction in headcount in some stats teams in our larger accounts. With this background one idea to explore with "gate keepers" is how BayesOpt helps the stats team enable scientists and engineers to deliver greater organisational value and help their business understand the value that the stats team contributes. Doing something that improves outcomes now is better than striving for perfection (that rarely happens).
Re: Learn from Feedback from our JMP Pro Workshop with Chris G. in HeidelbergHi,
after having some discussions withSenior Management (Head of MSAT, Director of the Site, and similar) one important use case that resonates very well is when the customer has to work with customers who are not happy to do a full experiment when developing/running a process for them. In all of my meetings they were asking a lot of questions and nodded when I asked if that would help them compared to what they currently are doing.
I thought it might be of interest for you all
