608175_en-US

liliconcom

Isolation forest and isolation trees

Isolation Forest has been emerging as arguably one of the most popular anomaly detectors in recent years due to its general effectiveness across different benchmarks and strong scalability. It is computationally efficient and has been proven to be very effective in anomaly detection.

 

Isolation forest compared to others (src)

The algorithm isolates observations by randomly selecting a feature and then randomly selecting a split value between the maximum and minimum values of the selected feature. The implementation is therefore the same as partition trees and bootstrap trees with the difference that the target function should be uniform (trying to split variables with a constant Y value).

Yet, if you try this in JMP, it doesn’t work.

By including Isolation Forest in JMP, similar tu predictor screening, users would have access to a powerful tool for detecting anomalies in their data. This could help them identify unusual patterns or behaviors that may warrant further investigation.

 

This paper studies how IForest works and improves upon its few limitations (i.e., extended isolation forest)

https://hal.science/hal-03537102/document

 

Scikit-learn documentation

https://scikit-learn.org/stable/modules/outlier_detection.html

 

 

Basic Data Analysis and ModelingPredictive Modeling and Machine LearningRe: Isolation forest and isolation trees

Thanks, Mia. Notice that the implementation of an isolation tree is the same as a partition tree. 

The "only" difference is that the partition happens randomly.

 

In JMP, if you introduce a Y without any variability (all 0s, for example), JMP doesn't perform any split (something expected, otherwise solutions will be stochastic at a certain point).

 

In the literature, this is known as Extratrees.

https://en.wikipedia.org/wiki/Random_forest#ExtraTrees

Re: Isolation forest and isolation trees - Status changed to: Investigating

Thank you for this request @FN . We are currently investigating. 

Re: Isolation forest and isolation trees

Extended trees or forests will be even better, indeed. 
Following your link, the original paper for extended is here: https://arxiv.org/pdf/1811.02141.pdf

Re: Isolation forest and isolation trees

Great suggestion @FN.

 

In addition to Isolation Forest, 

it might be also interesting to consider the Extended Isolation Forest instead of the "regular" Isolation Forest algorithm, as it provides more flexibility in the directions of decision boundaries (not only horizontal or vertical).

Here is an article explaining how Extended Isolation Forest work and how to use it : https://link.medium.com/TL1Lxirf9pb

Re: Isolation forest and isolation trees

Hello @mia_stephens, any update on this? Thanks

Re: Isolation forest and isolation trees

Hi @FN , we've done some research on using the Python integration in JMP 18 with the Python ‘isotree’ package, using JSL to create dialog windows in JMP. It looks promising, but we don't have anything to share at this point. Curious if you've taken a look at this approach?

Re: Isolation forest and isolation trees

Thanks, Mia.

 

As shared in my previous comments, this approach will be very useful in JMP as it outperforms other anomaly detection algorithms with global outliers.

 

KNN distances (within the outlier analysis menu of JMP) work well for local outliers.

 

Adding Ex. Isolation Forest there will be useful to quickly identify and remove all types of outliers in massive datasets.
Unsupervised Anomaly Detection Algorithms on Real-world Data: How Many Do We Need?

The partition tree can also be modified to split the data randomly when Y is not given.

 

 

 

Re: Isolation forest and isolation trees

Coming back to this request to see if there are any updates.

This paper summarizes why JMP should have this implemented.

“33 unsupervised anomaly detection algorithms on 52 real-world multivariate tabular data sets, performing the largest comparison of unsupervised anomaly detection algorithms to date. On this collection of data sets, the EIF (Extended Isolation Forest) algorithm significantly outperforms the most other algorithms.”

 

Re: Isolation forest and isolation trees

I just saw a JMP-Python integration webinar by Ross M (which was great) in which he highlights this approach in Python, but it seems like it could be relatively easily be done natively in JMP. It appears to represent a mix of techniques that JMP already does, like partition trees and multivariate outlier analysis.

Re: Isolation forest and isolation trees - Status changed to: Yes, Stay Tuned!

Under development as a Pythonic add-in.

Re: Isolation forest and isolation trees

Great news! thanks Mia  

Re: Isolation forest and isolation trees

Add-in is available in the marketplace now!  https://marketplace.jmp.com/appdetails/Isolation+Forest

Thanks @russ_wolfinger for providing that.  

Re: Isolation forest and isolation trees

Yes indeed, the add-in is now available. Thanks @MathStatChem for getting it started!

Re: Isolation forest and isolation trees - Status changed to: DeliveredRe: Isolation forest and isolation trees

@mia_stephens, thank you for the effort put into developing this add-on!

That said, I’ve noticed that its reliance on Python libraries makes it quite challenging to use on Windows machines at least. 

If there are no plans to implement this functionality natively in JMP (which would be fantastic, in my opinion :smiling_face_with_smiling_eyes:), would you consider open-sourcing the add-on? This way, the community could help maintain and enhance it in the future.

---

For example, I am unable to install isotree:

Python Install Packages("isotree");

 In JMP 18 I get this error:

Traceback (most recent call last):
  File "", line 198, in _run_module_as_main
  File "", line 88, in _run_code
  File "C:\Users\user\AppData\Roaming\JMP\JMP\Python\Python311\site-packages\pip\__main__.py", line 22, in 
    from pip._internal.cli.main import main as _main
  File "C:\Users\user\AppData\Roaming\JMP\JMP\Python\Python311\site-packages\pip\_internal\cli\main.py", line 10, in 
    from pip._internal.cli.autocompletion import autocomplete
  File "C:\Users\user\AppData\Roaming\JMP\JMP\Python\Python311\site-packages\pip\_internal\cli\autocompletion.py", line 10, in 
    from pip._internal.cli.main_parser import create_main_parser
  File "C:\Users\user\AppData\Roaming\JMP\JMP\Python\Python311\site-packages\pip\_internal\cli\main_parser.py", line 9, in 
    from pip._internal.build_env import get_runnable_pip
  File "C:\Users\user\AppData\Roaming\JMP\JMP\Python\Python311\site-packages\pip\_internal\build_env.py", line 19, in 
    from pip._internal.cli.spinners import open_spinner
  File "C:\Users\user\AppData\Roaming\JMP\JMP\Python\Python311\site-packages\pip\_internal\cli\spinners.py", line 9, in 
    from pip._internal.utils.logging import get_indentation
  File "C:\Users\user\AppData\Roaming\JMP\JMP\Python\Python311\site-packages\pip\_internal\utils\logging.py", line 4, in 
    import logging.handlers
  File "logging\handlers.py", line 26, in 
  File "socket.py", line 51, in 
ModuleNotFoundError: No module named '_socket'

And in JMP 19 Pro I get a compiler error.

//:*/
import jmputils
jmputils.jpip('install', 'isotree')
/*:
Collecting isotree
  Using cached isotree-0.6.1.post8.tar.gz (300 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: isotree
  Building wheel for isotree (pyproject.toml): started
  Building wheel for isotree (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  Building wheel for isotree (pyproject.toml) did not run successfully.
  exit code: 1
  
  [14 lines of output]
  C:\Users\user\AppData\Local\Temp\pip-build-env-e669figd\overlay\Lib\site-packages\setuptools\_distutils\extension.py:150: UserWarning: Unknown Extension options: 'install_requires'
    warnings.warn(msg)
  C:\Users\user\AppData\Local\Temp\pip-build-env-e669figd\overlay\Lib\site-packages\setuptools\dist.py:332: InformationOnly: Normalizing '0.6.1-8' to '0.6.1.post8'
    self.metadata.version = self._normalize_version(self.metadata.version)
  running bdist_wheel
  running build
  running build_py
  creating build\lib.win-amd64-cpython-313\isotree
  copying isotree\__init__.py -> build\lib.win-amd64-cpython-313\isotree
  running build_ext
  Compiling isotree/cpp_interface.pyx because it changed.
  [1/1] Cythonizing isotree/cpp_interface.pyx
  building 'isotree._cpp_interface' extension
  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
  [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for isotree
Failed to build isotree
error: failed-wheel-build-for-install

Failed to build installable wheels for some pyproject.toml based projects

isotree

0

Re: Isolation forest and isolation trees

The _socket error, you need to uninstall 18.0, and update to a newer 18.x.  And it has to be an uninstall, not a modify/restore. The socket shared library is not in the correct install location. 

 

For both 18 and 19, isotree is one of the finicky packages.  A workaround that usually works is to have a standard install of Python.org 3.11.x for JMP 18 or 3.13.x for JMP 19 and do a regular pip install in the default environment.  This usually builds properly.  Then do the jpip install from within JMP.  JMP can then pick up the local cached packaged build and then install into the JMP private site-packages directory.

 

Re: Isolation forest and isolation trees

Thank you for responding @Paul_Nelson.

Re: Isolation forest and isolation trees

error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
  [end of output]

When Python needs to build C/C++ based packages, you do need to have a C/C++ development environment installed.  Visual Studio Code should be sufficient.  This is no different than if you just had Python installed and tried to install isotree.