nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How do I auto-calculate precision and F1 score for models on imbalanced data?

Oleg

I like that JMP's prints a confusion matrix and ROC for the logistic (Generalized) regression, but some important metrics seem to be missing from the results: precision and F1 score. Is there a way to auto-include these in the output? Or, have it be computed as "automatically" as possible when I re-run a new classificaiton model. I am trying to avoid manually recomputing these with each new model.

UPDATE: Please post answers to the questions only. I am not looking for an opinion about whether I shall or shall not use Precision and F1. I am confidently looking for a way to compute these automatically in JMP. If you are not aware of such technique, there is no reason to contemplate whether this is doable or not, or post anything below. Thank you

Screen Shot 09-27-17 at 09.36 PM.PNG

Accepted answers

All comments

txnelson

If JMP does not provide a specific statistic you want for a given analyses, my experience has been, the developers at JMP have a good reason for having not included it. This Discussion Group, while monitored by JMP, is not the best way to get your statistic included into a future release. You need to send an enhancement request to

support@jmp.com

They have a formal tracking system for such items.

But now to the beauty of JMP. Many times, the Discussion Community will have a solution to what you need. But, since this topic has been discussed before, with no response on F1 etc. I would assume the solution is not available. However, many, if not most, of these requests can be added to your analysis through JSL. So in response to your question, I am sure you could write a script to add these to your analysis.

Kevin_Anderson

Hi, Oleg!

The F1 scores (and other "important" popular measures of accuracy) have been deprecated in some corners of the Machine Learning community for their performance under bias. Please reference the attached file, which suggests the utility of the Receiver Operating Curves and some other measures.

And like Jim notes, you can script anything your heart desires.

Powers Evaluation ROC JMLT2011.pdf

Oleg

Hi Jim, your answer implies that JMP is perfect and contains just what it needs to have (up to a good reason). That is, anything missing in JMP does not have a good reason to be in JMP. An interesting claim The discussions themselves generate plenty of "good reasons" for new JMP features. A question was posted about F1 (not precision) and still remains unanswered, yet, these measures are widely discussed and used in scientifica publications.

I like your suggestion of emailing the support, but they ask to launch a discussion. My hope is that the community will provide the answer, even if not right away.

If you feel my question can be improved, I welcome your suggestions.

Oleg

Hi Kevin, Thanks for the reply. I do not understand how others' preference for the use of F1 score shall affect my need for it in this discussion? Each and every metric has a bias. There are thousands (more recent) publications praising the value of F1 and precision metric.

The bottom line, in this discussion I seek assistance in auto-copmuting F1 and precision scores. Yes, it can be scripted; and I would greatly appreciate an assistance from someone who has already implemeted it, so that this wheel is not reinvented.

Indeed, I am alreday using ROC, but I also need F1 and precision. At the very least, the answer (whenever discovered) to the question I asked shall help others to auto-compute not just F1 or precision score, but any metric. Please keep the bigger picture in mind

txnelson

My previous response was intended to be much simpler than it appears you have interpreted it.

1. JMP is not perfect, but it is pretty good, and the developers really keep up with the areas they are working on, and therefore what they are presenting deserves a good look.

2. Best way to make sure your request for an enhancement has made it to JMP is to send it to

support@jmp.com

3. The request for F1 was already covered in a recent Discussion track, thus I assume there is not a user with a ready made solution. Therefore, my encouragement that it can still be done in JMP, where you could code the solution using JSL

Oleg

Jim, here is how a discussion spawning "good reasons" for new features to be added to JMP.

Kevin_Anderson

Hi, Oleg!

Here's the big picture: Jim Nelson has solved many, many problems in this community. Please don't force your opinions on him.

And here's my opinion: you may get more help with your scripting if you act like you've given it a try and ask for help with a little less arrogance.

Oleg

Hi Kevin. I appreciate Jim's help and have given him Kudos and accepted his solutions whenever it was appropriate. However, I must not agree with him on every point. In this thread, I expect answers to the questions asked, not opinions on what's fashionable in the ML world.

Oleg

Thanks Jim.

1. I agree

2. Support asks to use the discussion group, which is what I am trying.

3. Sure, JSL is a good direction. I was hoping that someone has already implemented it and can lead to an example.

Oleg

Kevin, you may have misread my original question. I was hoping for a built-in computation of both F1 and precision. JSL is a reasonable alternative.

Do you have an example I can follow?