840738_en-US

liliconcom

Predicting rare events

I am working with a customer on a Pro trial. They are working out of a ~3mil row dataset trying to build a predictive model to predict failures, which occur 0.1% of the time (binary response, continuous factors). Failures are killers and would gladly accept a model that can capture said failures even at the cost of losing passes. 

I have thrown just about everything I know how to do. To keep run times down they have subset the original file down to anywhere from 1-10% of the original data, trying various weighting techniques, Torch, XG Boost, and Model Screening. Boosted Tree has provided the greatest success but in each instance the model developed on the subset is too overfit to be useful on a either a similarly sized but uniquely different data set or the larger data set. 

Should I weight more (currently weighting pass/fail at 1/100), create a subset of data where failures are over represented? Are there hyperparameters that can be optimally set for this sort of highly imbalanced model? I'm looking for advice/guidance on what I might try with this person. Thank you.

Re: Predicting rare events

Have you given the Imbalanced Classification Add-In a try? I played around with it a while back and found it to boost performance on one of our sample data sets (which one now escapes me). I'm not expert in SMOTE or Tomek Links but I found the introduction of this Kaggle post to be helpful in understanding the very basics.

It should be a good next step if you haven't yet given it a go.

Re: Predicting rare events

These rare events make me think of Reliability, but I'm not sure that's the way to go.  So you've got continuous input factors over time, and every now and then you get a failure.  How about some of the control chart tools, particularly multivariate control charts?  Try to correlate when some combination of factors occurs with the failures.

Re: Predicting rare events

Have you looked at the Decision Threshold in Model Screening? It's used in binary classification. You can see how different thresholds, .1 vs .5 impact the model. 

Re: Predicting rare events

Have you looked at using Zero Inflated Binomial in the GenReg platform? Also, have you considered Decision Thresholds that result in higher Type I/False Positives?

Re: Predicting rare events

Also, while I haven't watched it yet, if I'm guessing correctly, @Jason_Wiggins 's recent Mastering Webinar on defect count models might be helpful: Making the Best Choice for Defect Count Models

Re: Predicting rare eventsZero inflated in gen reg sounds like a good approach but not sure about run times. Whichever model you end up with, be sure to use Decision Threshold to determine an appropriate cutoff (otherwise prediction formula assumes 50:50 for either event). I will send you a slide deck I put together to help decide on cutoff based on Decision Threshold report.Re: Predicting rare events

So... I don't know that 3M records should be too bad if they're running the pro based models on a decent machine (and not going crazy with the number of factors).  I think you've got a few problems here to consider.  

First, have they done some factor screening? Boosted tree or random forest should tell you if there are factors in there that don't matter.  Getting those factors down will reduce the fit times.  

Second (or maybe tied for first - I'm not sure), are they doing a stratified cross-validation? Have them use the validation column set-up tool to create a TVT cross-validation with the response in the stratification field.  That will keep the model honest.  I like to have a lot of data in the training group and decreasing amounts in the latter two.

The model types and distributions suggested are all good ideas - my bias for something like this would be towards the tree-based methods (Random Forest, Boosted Trees, or XGBoost) with a random seed for reproducibility.  They're designed for these larger data sets. I think Jerry's got an interesting idea with the rare event reliability angle.  Also you might consider this: https://youtu.be/Ac6E7da9igU 

On weighting - I don't think you're going far enough.  Using your 0.01% failure rate, I'd give the passing data a weight of 0.0001 and your fails a weight of 0.9999.  We had a spell a while back where this was coming up and the (tribal - I'll admit) guidance was to put the response column into distribution and reverse the proportions (what I describe in that second sentence).  It seemed to work pretty well.  

Last, maybe just as a recasting of the situation, evals are smoother when they are about getting them confidence that they can get to the solution to their problem.  They're less about actually getting them over the finish line for their problem. I think we both know something like this is going to take some cycles of learning on their part and will probably go well over the time remaining in the eval. If you focus on getting them to be confident they'll get to a solution with Pro they'll buy even if you don't get "the model" for their problem.  

Re: Predicting rare events

Thank you all for the input (this far)! I've thrown a number of the ideas listed below at this problem but not all of them and not in the ways recommended. I appreciate y'all's advice.