file:///C:/Users//AppData/Roaming/SAS/JMP/Addins/com.jmp.xgboost/XGBoost%20Add-In.pdf
Page 5:
XGBoost > XGBoost
The second level menu entry name is redundante and confusing.
I would suggest something that clearly indicates the action about to be taken, such as "fit model".
Page 6:
"If you want to perform a single holdout analysis instead of k-fold, change the type of the Validation variable to Continuous and make sure it has values 0 for training and 1 for validation."
It is nice that there is a way to make this work, but it is undiscoverable as is. This should be made explicit in the UI. A checkbox control for example.
Add-in UI
I think it would be a lot better for the user if the Add-in added new menu entries under the Analyze menu, along with the currently existing algorithms. I mean, XGBoost should be under Analyze > Predictive Modelling, after Boosted Tree. I understand this is more likely a limitation of the JMP menu customization system exposed to add-in authors.
XGBoost UI Model UI
The LRT has a "Generate Python Code..." option, which is the same name used by the published model in the Formula Depot.
However, the generated code is markedly different, and there is no clear indication of why a user should use one over the other. I suggest using different names and an explanation in the tooltip text of how they differ.
Different names will also make it easier to find them in the Scripting Index, where additional information about their differences can be provided.
Add-in Python Generated code:
xgboost_jmp.py
"# Please send comments, suggestions, corrections to russ.wolfinger@jmp.com"
It would be better to use a corporate account, such as tech.support@jmp.com, as opposed to an individual account.
import pandas as pd
Although Pandas is a good choice as far as a generic tabular data manipulation library for Python, assuming it precludes the use of new libraries that provide higher performance and scalability, such as cuDF and Dask. Coding against an adapter interface and providing a default implementation for Pandas would allow users to benefit from a default implementation while still keeping the door open on other deployment options. It shouldn't be hard as these new libraries try to follow the Pandas API as much as possible, sometimes even providing drop-in replacements. In particular, a POC showing our generated code scoring a Dask dataset larger than memory can hold, using multiple threads or perhaps even distributed computing across a cluster would be a great selling point for how JMP allows users to leverage the Python ecosystem with minimum effort.
import seaborn
import matplotlib.pyplot as plt
Adding code to visualize the results is nice, but it is dead weight for a deployment scenario. I would provide that functionality in a separate module and keep the deployment code as light as possible as far as dependency goes.
I mean, data visualization is nice for a demo but in production, it is just not useful. The generated code should be optimized for production scenarios, IMO.
def feature_name_replace(f)
This method can map two distinct feature names to the same name, generating naming conflicts with potentially bad consequences. For example, feature_name_replace("A(1)") == feature_name_replace("A.1_") == "A_1_".
The calling code does not check for such a conflict.
It might be unlikely but since we don't control how the user will name its variables, we should check for it.
def main(mname, path, df, yvar, xvar, wvar, vvar, cvar, parms, iter, action='fit',
verbose_eval=100, verbose=True, plot=True):
# parms: xgboost parameters, only those different from defaults are needed
There is no documentation for the valid or expected contents of the dictionary 'parms'.
Maybe a pointer to where they can be found?
# action='fit': fit xgboost models with repeated k-fold cross validation
# action='predict': predict new data, binary model objects must already be saved in mod subdirectory
There is no check for action='predict', it appears the main function will always attempt to evaluate the model. Is that necessary in case the user only wants to fit the model?
Also, the code that would save the model to disk when action='fit' is commented out. So there is currently no way to re-use the fitted model later when specifying action=='predict'
# save model binary object
# xg.save_model(fname)
# print(fname)
# versbose_eval: how frequently to print xgboost iterations
Typo.
# plot: plot actual versus predicted when action=='fit'
See comments above about the usefulness of plotting capabilities.
Formula Depot Generated code:
I used the LRT option to publish to the FD and then used the FD option to generate Python code (see comment above about re-using the menu action name).
The generated scoring code file appears to be correct and has no placeholders. In theory, it would be ready to be deployed. The problem is that like similar models generated in the field by our customers (link) the model is too large and crashes the Python interpreter. The line count for a model generated for PowderMetarllugy using the default values as suggested in the help is close to 82K lines of code.
There are many possible ways of addressing this issue, from reducing the number of temporary variables to splitting the generated code into multiple, smaller files (or at least functions). There is already a defect opened for that but it is not a priority.
Besides generic optimizations for large models as described above, the particularities of the XGBoost model such as avoidance of missing values and use of one-hot encoding could very likely be leveraged to generate a leaner representation, by avoiding some of the checks and transformations present in the generic code generation engine.
Scoring Code Validation
Assuming we can get models small enough to run without crashing the Python interpreter, have we compared the scoring results of the Formula Depot generated scoring code with the scoring results of the binary model generated by the Python wrapper? I think it would be important to know that the generated numbers are indeed the same. There is a testing tool built for this type of validation stored in Perforce at
//depot/main/test/Scoring/Python/.
I believe it could be used with very few changes if any.
xgb_jmp_m1.py
This seems to be the POC file showing how to use the library.
A Jupyter notebook would be a better choice, as it is the most common vehicle for this kind of "here is how to use my library" scenario nowadays.
It would also be a better place for the plotting code that is currently part of xgboost_jmp.py
