Bayesian approach to machine learning for Bernoulli response

Question

I have a data set where I have about 2000 observations of about 150 features or covariates.  I'm trying to predict a 0/1 outcome.  The outcome is a risk score for disease, R1.  The assay generates all the feature values (F1, ..., F150) in a database, as well as a risk value for a related disease (R0).  The thought is that the R0 is highly predictive for R1, and that if we use the features F1, ..., F150 we might have some sort of update rule we can apply to R0.  Stated a bit differently, I'm thinking of using R0 as the Bernoulli parameter for a prior distribution of R1 and somehow training an update formula to use on R0 based on the values of F1, ..., F150.

The training algorithm would have to allow a 0/1 probability score model to be built that allows us to specify a prior value for the score.  This sounds like a pretty straigthforward Bayesian problem.  But I haven't done this before and I'm not sure how the feature information would be incorporated into the parameter update expression.

Would we generate a second probability score?

Would we use the linear predictor prior to going into the logit transform?

Would we modify R0 in some additive way, or by some multipicative factor?

Clearly I have more questions than answers.  If anyone can point me in the right direction that would be great.  If JMP allows this sort of thing, I'd love to hear about it.

dale_lehman · Accepted Answer

I'm a bit confused - you say your model uses 100% of the features (150 of them!). Virtually all machine learning models I have run never use all the features. You may enter them all as possible, but rarely (never) do they all get used.  In any case, if you think your model is too complex, you can drop out each feature (tedious, but someone can probably help script this) individually to see how much the performance goes down. An alternate version of this - which I have seen in other software but not in JMP yet - is to replace each feature (one at a time) with random data and compare the performance.

You also appear to want to see if you can get better predictions by adding something to the model you already have (what is left to add if all the features are in it already?).  I think there are many ways to do this - but I think more about trying different models, and using some kind of ensemble method to combine them, rather than using additional features. Bayesian model averaging is one way to do this. But your original idea about using your first model as a prior and then adding features to produce a "Bayesian update" still doesn't make sense to me.

gene · Accepted Answer

Yes, 100% of the features are used to calculate p0.  The system actually generates thousands of features, but we needed to be able to validate the algorithm for computing p0 offline.  So we found (through reading the code) every feature that was used to calculate p0 and saved them.  Mind you, the "calculation" for p0 as a function of the features is too complex and implicit to ever let you write it down (hundreds of binary classifiers in parallel and serial configurations,  I didn't build the system).  But we do know that we have captured all the necessary and sufficient feratures to do an offline simulation and validate the computation for p0.

My goal is not to modify anything about the model for calculating p0.  It is to see if, by chance, the very same features might hold contain some "magic" information that allows us to predict some other disease.  My thought is that is unlikely, especially since the features have been generated specifically to, in some optimal and minimal manner, predict D0.  There are probably some information theoretic ways to test this asssumption.

Again, I need to come up with a fair and principled wat to either show that the concept just will not work or that it will.  It has been easy to generate plenty of red flags (covariate drift as an example).  My attempt to use some sort of modifed "Bayesian" approach is my way of using the best statistic the features have been designed to generate, and then see if there is some independent, residual, information that might help predict D1.  I'm a skeptic, but I'm among a group where others (management) have a dogmatic view that machine learning can magically solvce any problem imaginable.  They don't appreciate that most of these models (LASSO, ridge, PLS) are just, at the end of the day, just simple main effects (generalized) linear models.

Once again, I appreciate your engagement.  It has been very helpful.