nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Predictive Modeling with asymmetric cost / imbalanced data set

Nayimoni

Hello,

How can I instruct predictive modeling platforms on JMP Pro (partition, random forests etc.) to assign a higher cost to falsely predicting the minority class. I have a dataset that is highly imbalanced with a high cost for false negative (readmission rate of acute patients) and when I use the classifiers the AUC is low and all positive cases are falsely classified.

Please help on tools available to address this issue.

Thanks

Accepted answers

dale_lehman

As has been pointed out, improving the AUC and incorporating asymmetric costs/benefits are two different things. The AUC has to do with the ability of the model to classify correctly, while the profit matrix steers the classification errors towards where they do the least damage. If the AUC measure is not bad but the classifications are making costly mistakes (like failing to classify any of the 1 values), then you should try a different cutoff probability for the classification - that is what the profit matrix will do automatically, but you can also change the cutoff probabilities by hand to alter the misclassifications. If you still don't like what you can get, then you want a better model - one that will produce a higher AUC. Nothing about the misclassification costs will raise the AUC - only a better model can raise the AUC. So, if you need a better model, then try other techniques, transforming your variables, changing the model settings, etc. but don't use the profit matrix to try to improve the model itself.

All comments

ian_jmp

Take a look at the profit matrix (and links off this).

Nayimoni

Im not sure I understand how it works. My understanding is that the profit matrix appears after I run the model. I want to be able to specify a higher cost to the minority class and make sure the model takes into account that when building the prediction.

Can you elaborate on the procedure to achieve that?

Thanks

Nayimoni

Any help on this please?

dale_lehman

The profit matrix can be defined as a column property prior to building any predictive model and should do what you are asking for. However, I would note that I have had difficulty filling in the profit matrix - the example in the JMP help guide is clear but often does not match a real situation. If you decision problem is such that you can specify exactly what the cost of false positives and negatives are, then the profit matrix should work. For example, the JMP example is for an agent making airlline reservations, where a wrong prediction carries a known additional cost to the agent. On the other hand, if you are predicting something like customer retention, then the costs of a false prediction are not so clear. If you wrongly predict a customer will be lost, then you presumably will incur extra costs to try to keep the customer. For some customers this can turn them from lost to retained, but certainly this will not be 100% effective. It becomes complicated to figure out what to put in the profit matrix in such cases - and I would recommend not using that feature. Better to look at alternative cutoffs and confusioin matrices.

Nayimoni

Thanks. I tried it but the AUC of my model is still the same under different methods. It did not force the model to assign a higher importance to the minority class.