As I've shown in previous blog posts, the advantage of using Python integration with JMP is that you can use Python packages to access features and analysis methods that aren't included in JMP .
This time, as an example, we will cover how to implement the density-based clustering method HDBSCAN using Python integration with JMP. While JMP has cluster analysis methods such as k-means, it does not have HDBSCAN built-in.
Furthermore, it can be combined with the dialog box introduced last time. The dialog box also allows you to specify variables (columns). In other words, you can select the variables to use for clustering on the spot, specify the parameter values, and then execute the process. You can use HDBSCAN with almost the same feel as the JMP analysis platform.
ダイアログによるHDBSCANの実行Running HDBSCAN via dialog box
What is HDBSCAN?
HDBSCAN is a method for finding clusters by focusing on the "density" of data . It considers areas where data points are densely clustered as a single cluster, and divides the data using areas of low density as boundaries.
HDBSCAN does not require you to specify the number of clusters in advance, as it is determined automatically from the data. Another major feature is its ability to separate outliers as noise. In clustering methods such as k-means, outliers are always classified into one of the specified clusters. In contrast, HDBSCAN treats points that do not belong to any cluster as noise.
Instead of specifying the number of clusters, there are two parameters that determine "how densely populated an area should be to be considered a cluster."
min_cluster_size : The minimum number of data points to recognize as a cluster.
Smaller groups than this value are not considered clusters. Increasing the value tends to result in fewer clusters because smaller clusters are no longer recognized.
min_samples : The number of neighboring points to refer to when measuring density.
The larger the value, the more conservative the density determination becomes, and the more noise (-1) is included . Think of it as specifying the strictness of outlier detection.
Python-based code (.py)
Before creating the dialog, let's first write the HDBSCAN part using a Python-based method (.py).
With the sample data "HDBSCAN_sample.jmp" (1,000 rows, 6 columns of continuous data, attached to this blog), which was created using random numbers, open, running the .py file (Example2_HDBSCAN.py) will execute HDBSCAN using "X1" to "X6".
The following is the main part of the code. It converts the JMP columns "X1" to "X6" into a pandas DataFrame (a tabular data structure), standardizes the data, and then runs HDBSCAN.
Pythonを起点としたコードPython-based code
*This code requires that scikit-learn and hdbscan be installed in the JMP Python environment beforehand.
When you run it, three columns will be added to the data table.
- Cluster : Stores the cluster number to which it belongs (nominal scale). -1 indicates noise (outlier) that does not belong to any cluster.
- Cluster_Prob : This represents the strength of belonging to the cluster (0-1). While it's often translated as "probability of belonging," more accurately, a value closer to 1 indicates being at the core of the cluster, while a value closer to 0 indicates being near the boundary.
- Outlier_Score : A score (0-1) that indicates how isolated an object is relative to its surrounding density. A higher score indicates a more likely outlier.
データテーブルにクラスタリングの結果を追加Add clustering results to the data table.
Exploratory analysis using JMP
From here on, we will use JMP's features. First, by color-coding the markers by cluster in the data table ([Rows] > [Color/Marker Coloring by Column Value]), you will be able to see the differences between groups at a glance in other analyses as well.
When we perform principal component analysis and look at the score plot, we can see that the three clusters are clearly separated, and the red (-1) points are scattered away from each group.
主成分分析とクラスタリングの結果Results of principal component analysis and clustering
Performing multivariate outlier analysis using [Multivariate] > [Multivariate Correlation] reveals that this data is a pseudo-simulation. The neat color separation in the Mahalanobis distance plot, according to row number, is because the data was generated in groups.
Mahalanobisの距離Distance of Mahalanobis
Use a dialog box to create a general-purpose tool (.jsl).
Up to this point, the code has fixed variables "X1" through "X6," and the parameter values are written directly into the code. To use it with different data, you need to rewrite it each time.
Therefore, we will use a dialog to allow column selection and specification of min_cluster_size / min_samples . Since dialogs can only be created with JSL, the following code (Example2_HDBSCAN.jsl) will be JSL-based. There are two main points.
1. Dialog section
- Col List Box( All, … ) : Displays all columns of the data table in a dialog box (list on the left).
- Col List Box( Numeric, … ) : This is the list that will receive the selected column (the list on the right). Specifying Numeric means that only numeric columns will be accepted.
- Button Box(" > ") : Adds the currently selected column from the left list to the right list.
// ダイアログの起動
nw = New Window( "HDBSCAN Settings",
<
<MODAL>
= 2 ), V List Box( H List Box( Panel Box( "Columns", srcList = Col List Box( All, width( 180 ), nlines( 12 ) ) ), V Center Box( Button Box( " > ", xList << Append( srcList << Get Selected ) ) ), Panel Box( "Variables for HDBSCAN", xList = Col List Box( Numeric, Min Items( 2 ), width( 180 ), nlines( 12 ) ) ) ), Panel Box( "Parameters", H List Box( Text Box( "min_cluster_size " ), mcsBox = Number Edit Box( 30, 5 ), Spacer Box( Size( 24, 0 ) ), Text Box( "min_samples " ), msBox = Number Edit Box( 10, 5 ) ) ), Button Box( "OK", selCols = xList << Get Items; mcsV = mcsBox << Get; msV = msBox << Get; ) ) );
</MODAL>
2. Receive the Python result and add it as a column.
ダイアログで選択した列やパラメータの情報を、Python Send() を使ってPythonに送付します。Python側では、この変数を使ってhdbscanパッケージによるクラスタリングを実行します。
Python Send( selCols );
Python Send( mcsV );
Python Send( msV );
この後の <EM>Python Get()</EM> は Python Send() の逆で、Python側の変数をJSLに取り込みます。これにより、JMPのデータテーブルに新しい列が作成できます。
// Python結果の受け取り、列の追加
labs = Python Get( labels );
prb = Python Get( probs );
outs = Python Get( outliers );
cCol = dt << New Column( "Cluster", Numeric, Nominal, Set Values( Matrix( labs ) ) );
pCol = dt << New Column( "Cluster_Prob", Numeric, Continuous, Set Values( Matrix( prb ) ) );
oCol = dt << New Column( "Outlier_Score", Numeric, Continuous, Set Values( Matrix( outs ) ) );
When you run this JSL code on "HDBSCAN_sample.jmp", a dialog box will appear allowing you to cluster only four variables, for example, "X2", "X3", "X4", and "X5".
By adding a dialog box, you can apply the same script to other data as well. Let's use the five consecutive variables "P1" to "P5" from the JMP sample data "Powder Metallurgy.jmp" and change min_cluster_size to 300 and min_samples to 30 to match the number of rows.
HDBSCANのオプション設定HDBSCAN option settings
Clicking the [OK] button will display the principal component analysis report and other related information.
別のデータにおけるレポートReports in other data
As in this example, computational processing using methods not available in JMP is handled by Python packages, while the interpretation and exploration of the results are performed in JMP. These operations are then grouped into a dialog box, allowing users to freely select variables. In this way, a general-purpose tool can be created that is usable by JMP users.
Summary of all 4 parts
Over the course of four articles, we've explained "what you can actually do" with JMP's Python integration features, using two concrete examples: data acquisition from the World Bank and clustering using HDBSCAN.
What these examples have in common is that features and analytical methods not included in JMP are left to Python packages, and the results are explored and visualized using JMP's data tables and reports.
Furthermore, by grouping the specifications of conditions and variables into a JSL dialog, it becomes a versatile tool that anyone can use repeatedly without having to rewrite the code.The biggest advantage of JMP's Python integration is that it allows you to broaden the scope of your analysis by combining the strengths of both Python and JMP.
Nowadays, you can simply prompt a generation AI to tell you what you want to do, and it will write Python code for you. We hope this series will be helpful as a hint for utilizing that generated code within the JMP framework.
Finally, I'd like to share a video (in Japanese) that summarizes the content of this blog series (all 4 parts) in about 12 minutes.
by Naohiro Masukawa (JMP Japan)
Naohiro Masukawa - JMP User Community
