alternative cut-off confusion matrix

dale_lehman

I have used the alternative cut-off probability confusion matrix add-in many times.  Suddenly it is producing nonsensical results on some data sets.  I verified that it still works as expected on an old data set.  Attached is one of the recent data sets in which it does not appear to be working.  The target variable is Stroke and I have a Validation column and a number of prediction model probabilities labeled with the probability of a stroke given the name of the modeling method (e.g., look at the column labeled boosted tree or logistic regression for 2 examples that are not working properly).  Strokes are very rare (around 5%) in the dataset and the probabilities derived from the predictive models are correspondingly small.  But when I use the add-in and a range of cutoff probabilities from 0.1 to 0.9, I keep getting zero predictions of stroke - which does not match up with the distribution of probabilities from these models (which, as I said, are low, but not zero).  Can anybody shed light on why this is happening?

Thanks.

Strokes.jmp

SDF1

Hi @dale_lehman ,

 

  I can confirm that the alternative cut-off confusion matrix add-in doesn't work on the prediction columns you mentioned in your post. However, I re-ran the Boosted Tree platform following the same predictors and response that you used, even verified the equation was the same, but the alternative cut-off matrix add-in worked correctly on this new formula. I found the same thing with the fit nominal logistic. If I re-run it (I do end up with a different equation), I can run the alternative cut-off and it works as expected. I'm running JMP Pro 15.2.1. Something might be a bit off with your two fits. Have you checked the ROC curves for the fits to see if they're above the diagonal line. There was no saved script for those two model formulas in your table, so it's hard to assess how well the fits work. Have you tried re-fitting the logistic and boosted tree models again to see if the newer equations work? When I run them a second time, it works with the add-in.

 

  On another note, I noticed that you used a random seed in generating your validation column. Since you have very few instances of stroke==1, you might consider making a validation column that is stratified on stroke so that it can take this into account. Also, when performing the nominal logistic fit, it might be better to model the Target Level: 1 rather than 0, since you're probably trying to predict the likelihood of a stroke (stroke==1) versus the likelihood of no stroke (stroke==0).

 

  Just some thoughts. Hope this helps.

 

Good luck!,

DS