Bayesian Optimization GP vs standalone GP

matteo_patelmo

Hi,

I'm trying to understand the behavior of gaussian process models in the "gaussian process" vs "bayesian optimization" platforms (JMP Pro 19.0.1 )

Fitting these simple simulated (deterministic) data: 

With Gaussian Process:

with Bayesian Optimization

 

Why is the confidence region for the BO model not shrinking to 0 at the data points, like I see in the GP model?

 

thanks

Matteo

 

ps.

This is the formula for Y:

4.3 + -2 * (:X - 1) ^ 2 + Exp( -3 * (:X + 3) ^ 2 ) + 3 * Sin( :X + 2 )

 

 

 

 

 

 

Victor_G

Hi @matteo_patelmo,

 

For the moment, the technical documentation for Bayesian Optimization is quite limited, and some technical details are not available regarding the calculation of confidence intervals/error intervals. There might be several aspects leading to the differences you see between the two platforms :

User inputs:

• The adequate use of the Continuous Correlation Type for your dataset (more correlation types are available in Bayesian Optimization than in GP (only Gaussian and Cubic)),
• The use of "Estimate Nugget Parameter" in GP, which enables the prediction model to smooth over the noise instead of perfectly interpolating. This option is not by default in classical GP, but the nugget parameter seems to be estimated by default in OB, which can increase the differences (but not in your example with a perfectly known formula).

JMP platforms inner workings:

• The fitting (and validation procedure) of classical GP vs. GP from OB.
  • In classical GP, the fitting and validation procedure rely on Jackknife method, where you have a Leave-One-Out procedure where you calculate your metrics on the samples that are kept (training samples), and then average the results for each LOO samples. 
  • In GP from OB, the fitting and validation procedure seems a little different and seems to rely on a classical LOO procedure (not 100% sure), where the metrics are calculated on the excluded samples (test samples). This could lead to an increase of uncertainty shown in the Profiler and different predicted values.
    You can see the fitting is different between the two platforms as if you're launching your dataset with the same correlation types and estimating the nugget parameter in both platforms, the values for theta, nugget parameter and GP variance are different.
• The calculation of confidence intervals : 
  In classical GP, the display of confidence intervals in GP is done using the variance formula of GP and the quantiles How to save prediction/confidence intervals from Gaussian Process model? The variance formula uses the theta value, and a distance between each point in the range to the closest point measured. Hence the "zero uncertainty" at the location of already measured points.
  The GP from OB provides a "Measurement Error" : "The Measurement Error is calculated as the product of the nugget parameter, τ, and the Gaussian Process variance, σ2". It seems the measurement error is also used, in order to avoid a "zero uncertainty" situation in the area where you have already sampled, in order to reflect experimental error.

Of course these interpretations have to be confirmed by JMP technical staff, these are just my impressions looking at the working of the two platforms and the outcomes at the moment.

 

Hope to get the inputs from JMP Technical staff to better answer your interesting question !