Integrating an initial estimate (heuristic) into a predictive model (machine learning)I'm trying to develop a tool I call Blend Search. It works OK, but I envision an improvement that I don't yet know how to implement technically.
What Blend Search does:
>> input is a collection of components with their properties
>> generation of a composition matrix
>> application of heuristics (simple models) to calculate the properties of blends
>> application of a more complex model, currently a Boosted Tree. Its predictors are the properties of blends estimated by heuristics, and its output is also a property of the blend
I created a visualization of Boosted Tree (see attached video)
One of the predictors (Predictor 3) already provides an initial estimate of my target property (Predicted). In my initial idea, the other two predictors were intended to refine this first estimate. Today, the model provides consistent predictions; we rarely deviate from the predicted range.
But in reality, I'm not correcting my initial estimate; I'm correcting the average of the Y values in the training set (the default Boosted Tree intercept). If I want to scale up my Blend Search, this approach will no longer be sufficiently robust.
I'd like to be able to make predictions by correcting my initial estimate. For example, I'd really like to have a variable intercept type for my Boosted Tree. I can't quite picture the right solution, but I'm pretty sure some techniques already exist. If you have any ideas, I'd be very happy to hear your thoughts.
Predictive Modeling and Machine Learning
Re: Integrating an initial estimate (heuristic) into a predictive model (machine learning)Hello @RikkiTikkiTavi ,
I'm not sure I fully understand the algorithmic modeling chain, but I might have an idea for you. Two approaches are possible to improve predictions: a boosting approach (to reduce prediction bias) and a bagging/model averaging approach (to reduce prediction variance).
- Boosting Approach: If the goal is to improve predictions of the target property by correcting a bias (predicted mean response versus measured mean response), you could train a machine learning model to predict the residuals of this target property from the raw and/or estimated data, as well as from predictions of the target property (in order to model errors based on the original variables and predictions). This would give you a model that predicts errors in the boosted tree and serves as a correction layer for the boosted tree (by adding/subtracting the residual predictions from the boosted tree predictions).
I've used this approach several times, and the performance gains can be quite significant. It might be worthwhile to use a different type of model for error modeling than the one used for predictions (i.e., something other than Boosted Tree in your case). Also, be sure to thoroughly validate this new model, avoiding overfitting, with an appropriate validation strategy (for comparison with the Boosted Tree model alone) and testing strategy (for evaluating performance under real-world conditions). Avoid any risk of data leakage by only using data available at the time of the prediction for errors: predictors plus predictions of the target property.
- Bagging/model averaging approach: Another option to reduce the variance of predictions is to train a set of models in parallel (in addition to the Boosted Tree model) and take the mean or median of their predictions. With this approach, however, it's crucial to implement an appropriate validation and testing strategy.
The choice between these options depends on the performance of your initial Boosted Tree model and the noise in the data: if the Boosted Tree tends to overfit or if the data used is noisy, a bagging approach is recommended. If the Boosted Tree underperforms and has a high bias (which seems quite unlikely given the model's behavior, unless specific parameters are used), the Boosting approach will be more effective. I might prioritize testing the bagging-type ensembles approach, since the prediction model used (Boosted Tree) already employs a Boosting-type approach.
Hopefully these options will inspire you,
Re: Integrating an initial estimate (heuristic) into a predictive model (machine learning)Hello @Victor_G ,
I had already tried predicting the residuals (the difference between the actual value of my target property and its initial estimate). I obtained a slight improvement on the training set, but it failed on the test set (the RMSE doubled). My test set represents the scale-up I'm aiming for; it's a subgroup of very different compositions, but the properties of these mixtures are almost in the same range as those of the training set. My current Boosted Tree predicts fairly well on the test set.
The idea of model averaging is interesting. Do I need to find models that make sufficiently different errors to test the approach? I think I'll try bagging first. I know my data is noisy, and I imagine this type of model will also be easier to integrate into my Blend Search tool than a set of 2, 3, 4... different models.
Thank you very much for your ideas;
Re: Integrating an initial estimate (heuristic) into a predictive model (machine learning)Hello @RikkiTikkiTavi ,
Yes, the difficulty of this approach lies in the right validation strategy to avoid overfitting (improvement on the training set, but stable or even deteriorating performance on the validation dataset).
I'm a little concerned that you might have a domain shift based on your description of the test data: you risk having different relationships between inputs and outputs learned by the model during training than those existing for the test. Would it be possible to mix/ stratify the training/validation/test data to avoid this shift (and only have scale-up data in the test)? This would allow the model to have information about the scale-up as well, perhaps by indicating it in a new variable used during training (experiment_type: R&D / scale-up / Prod, for example; it would depend on whether the different stages have different intercepts and therefore require this variable).
Regarding the rest of the process, here are some remarks/ideas:
- Start by setting up the training/validation/test strategy , if possible by doing a stratified split with respect to the variables to ensure that the entire experimental space is seen in each of the sets (to ensure relevant training and good generalization of performance).
- Before training the Boosted Tree (or other model), set up comparative models: a benchmark model (I recommend the Random Forest ) to outperform in validation against the other models, and a model to assess the lower limits of performance of the tested model(s): for example, a naive model that only "predicts" the average response of the training set across all other sets (validation and testing). If the other models are worse than or equivalent to this model (for example, with an RMSE close to or higher than the naive model), this may indicate excessive noise in the data relative to the signal, a lack of information from the factors, and/or insufficient variability in the response.
- I would also recommend testing different models, for example with the Model Screening platform, by performing cross-validation only on the training set. From there, it is possible to select the most promising models and proceed.
- Depending on the models' performance, retraining on the training set is possible, allowing for comparison of their performance on the validation set to select the best-performing model. In parallel, it's possible to take the selected models already trained with the previous crossvalidation and perform Model Averaging using the Model Comparison platform. The interesting aspect of Model Averaging is to combine models with different "construction types" (inductive biases) – such as tree-based models like Boosted Tree, Random Forest, or XGBoost – with other models like multivariate regression, SVM, Gaussian processes, or small neural networks, depending on their performance. These models should have good individual performance (they must have captured the signal in the data) and, ideally, slightly different error rates (some overestimate certain observations while others underestimate them). Comparison of this macro-model with individual models retrained on the training set can then be done using the same Model Comparison platform by comparing performance results on the validation set.
- Once the best option is selected (macro-model or individual model), performance is estimated on the test set to evaluate the behavior of the model once deployed.
It is also possible to bootstrap on Machine Learning models other than the natural bagging of the Random Forest, to improve the robustness of models in training.
Hopefully these suggestions will help you.
