Generate Shapley values from Neural Net

Question

I am aware the ability to generate Shapley interpretations is integrated into SAS Viya; has anyone had success in generating these values in JMP? I Have a customer that is bouncing between NN in Pro and Python to build a model then generate SHAP values but would like to perform this interpretation natively in JMP or via an Add-In. Has anyone tried this? Is it possible?

Peter_Polito · Answer

Thanks for your thoughts @SamGardner. Thus far the customer has generated values for a linear model only but are unsure how to do so with a NN (they seem to be using this as a guiding document for their calculation: https://shap.readthedocs.io/en/latest/index.html).

SamGardner · Answer

In the case of a linear regression model (just main effects, no interactions or curvature), I think the Shapely Value would be directly proportional to the slope estimates for the factors in the model.   Shapely values are a measure of "how much did each factor contribute to the output mean").

When you have a more flexible model (with interactions and/or curvature, or something more wiggly like a Neural Net), what you have to do is determine the average marginal effect of the factor (taking into account all of the other factors).  Under the profiler, you can create graphs of this estimated average marginal effect, using the Variable importance plot, but it doesn't report the actual table of values used to make the plot.

I suppose you coulld create this manually using a simulation output from the Profiler, also, with some summarization across each factor.

If they can describe the methodology they use for determining the Shapely values, it could probably be scripted.