Imbalanced Classification Add-InThe Imbalanced Classification add-in features sampling techniques that attempt to impose a more balanced distribution between the two classes. The sampling techniques include the synthetic minority oversampling technique (SMOTE), Tomek links, and a combination of the two, as well as some basic sampling approaches. The Tomek Sampling, SMOTE Observations, and SMOTE plus Tomek options enable you to apply these sampling techniques on their own to support your specific modeling efforts.
The comprehensive Evaluate Models option, which requires JMP Pro, enables you to fit models using various sampling methods and compare them on a test set to select thresholds using Precision-Recall, ROC, and Cumulative Gains curves, as well as other measures of classification accuracy. The other three options do not fit models, but rather enable you to apply the Tomek, SMOTE, and SMOTE plus Tomek sampling schemes to your own data.
The SMOTE, Tomek, and combined SMOTE and Tomek sampling techniques use the concept of nearest neighbors. The add-in uses Gower distance as its distance metric, which allows for continuous, nominal, and ordinal predictors. These options do not require JMP Pro.
Note: All options require JMP version 15.2 or higher. Excluded rows and rows with missing response values are ignored by the add-in.
Version 2, released 3/25/2021, supports JMP 16 and improves the handling of rows with missing values for all predictors.
Version 2.1, released 10/3/2023, fixes a bug in the interpolation of P-R values that occurred for specific data tables.
Predictive Modeling and Data MiningStatisticsRe: Imbalanced Classification Add-InHello @michael_jmp
Thank you very much for this add-in. But for the moment it handles only binary categorical/ordinal unbalanced response ? Will it handle later on multi-class response ?
Best regards,
Re: Imbalanced Classification Add-InThanks for the feedback, and sorry for the slow reply. I don't think that the suggested change to nIncr is the right way to fix this bug. I am attaching an updated version of the add-in (ver. 2.1) that has a fix included. (The fix changes the condition in the If() statement. Basically, if the Recall values are close enough together, we don't need to do any interpolation and therefore don't need to go into that If() statement at all.)
Re: Imbalanced Classification Add-InHi @michael_jmp ,
One of my customers got the same Error message as Sandra1 ("Concatenate for strings or matrices only at row 21189 in access or evaluation of 'V Concat' …") when analyzing some large data set. After looking into the source code, I found that such error arises from the below part of the code:
The variable "nIncr" equals 0, so that J(nIncr-1,1, .) is now a scaler rather than a matrix, which causes the V concat operation to be invalid.
To solve this issue, I hard coded nIncr to be "nIncr = nRows(PrecisionTemp) - nRows(SensitivityTemp) + 1;" and can now run the add-in on large data set successfully.
The question now is that I am not sure whether such code modification is valid, since I didn't have enough time to read through the code. Could you please help to confirm that?
Re: Imbalanced Classification Add-InHi @Sandra1 ,
Yes, all operations in JMP are performed in-memory, so you are limited by the amount of memory available on your machine.
Re: Imbalanced Classification Add-InDear @michael_jmp ,
Thank for your kind response. Yes, it works if I reduce the size to about a tenth of the original data table. Is the operation running out of memory something to do with my computer hardware?
Re: Imbalanced Classification Add-InHi @Sandra1,
Thanks for trying out the add-in. If you reduce the size of your data table, are you able to get the add-in to work without running into the errors? My guess is that one of the operations is running out of memory on your machine for the number of observations that you have.
Re: Imbalanced Classification Add-InHi,
When creating a weight column using a total of 21188 pieces of data, an Alert will appear, as shown in the figure,
It was later found that it may be related to some problems when generating SMOTE plus Tomek? Therefore, after unchecking SMOTE plus Tomek, continue to try other model constructions, and the Alert shown in the figure still appears. The training set data was successfully generated, but there are many missing values.
Also, the Precision-Recall Curve window is not generated. How can we determine the source of this problem and what to do about it?
Thanks,
Sarah
Re: Imbalanced Classification Add-InHi @Lu - thanks for the suggestion. However, we don't have any plans currently to add models to the add-in.
Re: Imbalanced Classification Add-InHi, @michael_jmp, since XGboost is a ML model more frequently used since its high performance. Do you think you would be able to implement this Model type into the Add-in in the near future? Regards, LuRe: Imbalanced Classification Add-InHi @OneSidedLemur23 - I also worked on this add-in. If you are able to share your data (or a data set that is similar to your data), that would be helpful in trouble-shooting the issue you are having. If you are able to, please email me at Colleen.McKendry@jmp.com and hopefully we can get some answers for you!
Re: Imbalanced Classification Add-InHi, @michael_jmp , thanks for your add-in. There's a probalem when I use it. For example, I have a dependent variable y and some independent variables, when I use the "SMOTE observations", I draged variable y to "binary class variable" and indendent variables to "X, predictors", and a new dataset was generated. But in this new dataset, the values of indendent variables were empty, how can I add the new generated observations to original observations? Thanks!!!
Re: Imbalanced Classification Add-InI'm afraid it's the version 2 of the add-in. I also tried installing it again, but nothing changed at all. I am using JMP Pro 16, I'm not sure if there are some issues with this version either. Thanks!!
Re: Imbalanced Classification Add-InHi @Zuska-Ariri, thanks for trying the add-in. Sorry that you're seeing some errors. We did identify the problem that @Lu reported to us, and it should be fixed in the "Version 2" of the add-in, linked on this page. Can you confirm that you are using version 2 of the add-in? Also, what version of JMP are you using?
If you're still seeing errors with version 2 of the add-in, we can investigate further.
Thanks,
Michael
Re: Imbalanced Classification Add-InHi, thanks for the numerous clarifications in the video. But I have been having the same issue as @Lu , it actually happens no matter what data I use, I also tried using the sample used in the video but nothing changed. Has the source of this problem been identified so far?
Thanks btw
- Zuska
Re: Imbalanced Classification Add-InHi @bfrank, it's hard to state a specific limit on the volume of data that the add-in can handle, because it is mostly limited by the amount of memory on the current machine (similar to the rest of JMP). We do know that especially as the number of categorical predictors increases, the runtime increases. Sorry we don't have more definitive specifications.
Thanks,
Michael
Re: Imbalanced Classification Add-InMichael,
Thank you for posting this. Can you clarify what limitations exist for this add-in. It appears that it can't handle certain volumes of data.
Re: Imbalanced Classification Add-InHi @Lu, sorry to hear you are seeing an error message with the add-in. Would you be able to send me your data in a private message so I can investigate what might be causing the problem?
Thanks,
Michael
Re: Imbalanced Classification Add-InHi,
When running the add-inn I get the following JMP Alert "Invalid subscript (must be number of list of numbers) in access or evaluation of classvec[/*###*/near_row]', classvec[/*###*/near_row]
A pupup window "Creating weighting columns" shows up and the analysis is stocked.
I already reinstalled the Add-Inn without any help. Did somebody already had this problem? Any suggestion?
Thx in advance
Lu
Re: Imbalanced Classification Add-In@Raaed This paper from He and Garcia is a good overview of the sampling techniques:
He, H. and Garcia, E. A. (2009). “Learning from Imbalanced Data.” IEEE Transactions on Knowledge and Data Engineering. 21, 9:1263-1284. https://www.academia.edu/29164932/Learning_from_Imbalanced_Data
Re: Imbalanced Classification Add-InHi
please, the source of sampling techniques (books or articles)
Re: Imbalanced Classification Add-InGlad to see this add-in!
