Hi,
a customer asked why the Optimal CCC specified in the cluster comparison report for kmeans clustering sometimes is lower than the CCC of other clusters, which is contradictory to the documentation.
I could reproduce this behaviour using the nicardipine data set and kmeans clustering as attached (run the script for clustering)
is there another criteria involved in the decision not mentioned in the documentation or why does this happen?
I think that there may be several things that cause this anomaly. If one of these is causing it, the documentation should mention it, so I think more detail is needed in the doc.
My thoughts:
The nicardipine variables you used are really categorical, not continuous. That could be contributing to the problem. The discreteness of the data typically leads to the CCC choosing too many clusters.
Two of the variables are constant values. Even the doc says that they are ignored, so I don't think that is the problem.
Every cluster from 6 on up has a cluster with only a single observation. That may be why those clusters are not "optimal" (but I really don't believe this either, but it is possible).
Looking at the CCC across clusters, the CCC drops dramatically and goes negative. Negative CCC values usually indicate outliers. Because there are 5 cluster sizes in a row with a lower CCC, I believe anything with a higher CCC than the 3 cluster model would indicate an overfitting. Could JMP be applying a similar logic?
If you run KMeans with clusters from 4 to 12, instead of 3, you get the results you expect, so perhaps this is a bug??
For completeness, the JMP doc references a SAS technical paper, I have included the conclusion from that paper for completeness. The full doc is here: https://support.sas.com/kb/22/addl/fusion_22540_1_a108_5903.pdf
Certainly an interesting situation. I would love to hear the real reason for 3 being chosen!
Thanks @Dan_Obermiller. I will check with my customer for outliers, #values in clusters and type of columns used. This might shed a light to it.
But I hope someone might know the underlying reason (bug, expected, type of data, ...).
So, I now have more information on the customer's data:
613 observations, 39 variables [based on SAS Documentation one might be able to look at about 60 clusters]
- Are variables categorical/binary?
6 variables have about 85%-90% value 0.
8 variables have just 20-25 different values.
- How about outliers (negative CCC)?
Rough check using PCA and outlier analysis, removed most extreme outliers only. Used to check clusters anyhow afterwards using parallel coordinates plots, declare few as noise, few put together. So a optimal CCC is just a guideline and needs further look at the clusters. however it is hard to understand that Optimal CCCC is selected on an earlier cluster with not the highest value. In my case just the N=2 and N=3 clusters are negative, N=4 is optimal though N=7 and even higher N is higher value.
- SAre there clusters with single values involved?
Yes & No:
Method NCluster N Rows CCC Best
K Means Cluster 2 . -11,181
K Means Cluster 3 . -25,497
K Means Cluster 4 . 4,46481 Optimal CCC
K Means Cluster 5 . 0,87854
K Means Cluster 6 . 7,58562
K Means Cluster 7 . 4,0766 à one Cluster with 1 observation
K Means Cluster 8 . 25,9692
K Means Cluster 9 . 16,4746
K Means Cluster 10 . 23,1449 à one Cluster with 0 observation (y, 0…)
K Means Cluster 11 . 24,0572
K Means Cluster 12 . 26,9875
K Means Cluster 13 . 27,7786 à one Cluster with 0 observation (y, 0…)
K Means Cluster 14 . 29,0657 à one Cluster with 0 observation (y, 0…)
K Means Cluster 15 . 18,975 à one Cluster with 0 observation (y, 0…)
As of NCluster = 3 ther eis at least one cluster with about 10 observations
- How about using only 5-15 clusters?
Yes, then the Optimal CCC is correctly displayed!
!!! Also if I uncheck "use within-cluster std deviations“ and use 2-15 clusters the displayed result is as expected.
Method NCluster CCC Best
K Means Cluster 2 -8,3209
K Means Cluster 3 -14,199
K Means Cluster 4 9,28967
K Means Cluster 5 5,42936
K Means Cluster 6 24,6335
K Means Cluster 7 30,4458
K Means Cluster 8 23,5854
K Means Cluster 9 36,858
K Means Cluster 10 35,6172
K Means Cluster 11 36,5099
K Means Cluster 12 38,4017
K Means Cluster 13 48,2059
K Means Cluster 14 44,3545
K Means Cluster 15 49,7344 Optimal CCC
==> So it seems that there is a kind of standardization ongoing within the clusters that result in different Optimal CCC than not using this option. If that's the case, then the display should be corrected as the CCC seems to have then different values for the unchecked and the checked option, this should be mentioned somewhere in the report as well of course in the documentation.
Any insight by developers/testers/writers might be of value. I can of course also put that through tech support, if this is something that should be changed in JMP.
Thanks,
Martin
