Do you work on predictive modeling? Are you building models across different platforms and looking for an easier way to tune your models in hyperparameter space? If so, then this interactive GUI might be for you.
This Generalized Auto Tuning add-in is designed to work across four JMP modeling platforms: Neural Net (NN), Boosted Trees (BT), Bootstrap Forest (RF), and XGBoost (XGB). The purpose of the add-in is to help narrow down the hyperparameter space for each of these platforms by utilizing tuning tables that can search the hyperparameter space.
The add-in runs a model for each setting of the tuning table and records the statistics of the fit for later review by the user. It's highly recommended to re-run the fit with the standard JMP platform using the optimized tuning parameters obtained from this add-in. The user can then monitor model performance with JMP's built-in methods within each platform.
The following files are included for download:
- genat.jmpaddin: the Generalized Auto Tuning add-in installer for JMP
- PowderMetallurgy_mod.jmp: a modified version of the powder metallurgy JMP file that contains nominal, numerical, and ordinal Y columns along with a two-level validation column (train and validate) stratified by Y Numeric and a three-level validation column (train, validate, test) also stratified by Y Numeric. This file is to use as a practice (and check that the add-in works as expected).
- NN Tuning.jmp: this is the tuning table format for the NN platform
- BT Tuning.jmp: this is the tuning table format for the BT platform
- RF Tuning.jmp: this is the tuning table format for the RF (Bootstrap Forest) platform
- XGB Tuning.jmp: this is the tuning table format for the XGBoost platform
After downloading and installing the add-in, there will be a new drop-down option in the Add-Ins menu called "Generalized Auto Tuning".
Selecting the add-in will pull up a new window where the data table, tuning table, and modeling method can be selected. The Data Table is the table containing the response and factor columns, whereas the Tuning Table is the appropriate tuning table corresponding to the Modeling Method. Clicking the select table buttons will bring up a selection window to define which tables will be used for that purpose. The default Modeling Method is NN. Selecting the "Recall" button will set the Modeling Method to whatever was used previously (except when running it the first time).
After selecting the data and tuning tables, and modeling method, click "OK" to open up the next interface window where columns from the data table will be cast into selected roles Y, Response; X, Factor; Validation, etc. This window also allows for the selection of the appropriate Validation Method (based on the chosen Modeling Method). For example, the NN platform can accept Excluded Rows Holdback, Holdback, Kfold, and Validation Column as validation options. Each modeling platform in JMP will accept only certain validation options.
After casting the appropriate columns into their roles, and selecting the appropriate validation options, click "OK". This will start the modeling process based on the data and tuning tables fed to the add-in. Two new windows will pop up. The first is an output data table starting with the name "Output of " followed by the tuning table name, validation method, and the input data table name. This output data table will contain all of the input parameters to the model as well as all of the relevant output statistics, depending on the Response column data type: nominal, ordinal, or numeric. The other window that pops up is an Overall Modeling Progress window, which visually shows how far the overall process is with the tuning table (this could be hundreds or thousands of rows, depending on the tuning table).
After the add-in has performed all the modeling steps defined in the tuning table, the statistics can be compared with the tuning parameters. For example, the Generalized R2 for the training, validation, and test set vs N_Boosts, etc. Or, the table can be sorted by the statistic of interest. Optimal settings can then be determined by the user for going back and running within the standard JMP platform.
This is the first time I've attempted an add-in, and by no means is it bug-free. I did have to look up a few functions and used some scripts posted by JMP Staff, like how to get a "Recall" button to work. Please try it out and if you find bugs in the add-in, please let me know and I'll try to fix them ASAP.
Some NOTES about the add-in functionality:
The add-in does require prior knowledge from the user with regards to expected response column types (e.g. at present, the JMP BT platform can not take nominal/ordinal response columns with more than two levels). Or, that some platforms can only use validation columns with two levels, e.g. training and validation. The add-in is only compatible with JMP Pro v15.0 and higher (because of the two-layer NN and XGB platforms).
In addition, there are specific columns that are fed to the modeling routine from the tuning tables (tuning tables included). Values for the different "runs" of the tuning table can be entered manually or generated using the DOE platform in JMP. Please note that if using the DOE platform, when a tuning column is constant, the DOE platform won't include it in the tuning table. As a result, when the add-in starts the modeling steps, it will use the default value for the tuning parameter unless it is manually entered into the final tuning table.
Thanks for taking a look and trying it out.
Good luck!,
DS
