444726_en-US

liliconcom

Shapley values for predictive models

Understanding the contribution of individual independent variables in a predictive model  would be invaluable. At present, we can infer the contribution and/or effect of a variable on a model via the prediction profiler, which is great, but to be able to quantify the contribution--and to be able to quantify the contribution on a row-by-row basis, would be next level. This feature has been specifically requested by customers in the O&G industry but its application is ubiquitous. These values are easily acquired using tools like Python and R. To make them available in the point and click environment of JMP would further separate us from our competitors. I have had multiple customers ask about how to get individual contributions of independent variables in a quantifiable way, and several ask specifically for Shapley values.

Statistics and AnalyticsRe: Shapley values for predictive models

@mia_stephens the account I've been working with most recently is Continental Resources (Kendall Sizemore would be more than willing to chat about this) but I've had similar requests from other O&G customers (Murphy, Oxy) who are a bit more tight-lipped. They cite a lot of work by Novi Labs, which is a ML service company in the O&G industry. The method we can currently run in JMP was developed by @kemal_oflus--where you use the scaled estimates from a linear model predicting the divergence of a ML model (the difference of average predicted response from ML model from the actual responses) using  standardized predictors. You can plot the scaled estimates in GB to give you a 'weight' of a given factor on a response. This ultimately tells you  whether a given factor positively or negatively affects the model (which is great) but it cannot be broken down to a row by row effect nor can it be applied directly to the ML model because scaled estimates are not available in NN, BS, etc.

A great explainer that includes the math: https://christophm.github.io/interpretable-ml-book/shapley.html

Shapley explainer: https://www.youtube.com/watch?v=-taOhqkiuIo&ab_channel=MicrosoftDeveloper 

A few screen grabs from Kemal's procedure: 

scaled estimates from a model built on standardized factors predicting average ML model divergence

divergence of a single factor (a Frac Fluid parameter)

 

 

 

Re: Shapley values for predictive models

@Peter_Polito and @nick_shelton, do we have any named customers/accounts we can list, or customers we can speak with (if needed)?

Peter, can you please share the method you developed, and your example output? Also, favorite references on the topic if you've got any.

I will add this information to the Jira.

Thank you!

Mia

Re: Shapley values for predictive models

Shapley values are quickly becoming the standard for interpreting feature importance and effect size in machine learning (ML) models. An emerging trend in ML is to move away from their black box reputation and create models that are interpretable and trusted. Tools like Shapley values offer the ability to be more transparent and confident in the model's predictions. This capability should be available for all of JMP Pro's machine learning models.

Re: Shapley values for predictive models

An example of how this would be used: A customer is using JMP Pro’s boosted neural net to construct a ML model. Where they are limited is their ability to quantify the effect of a given independent variable on the response. For example, if one analyzes the production values of a three wells they might find one is performing well because it is in a prime location and they constructed a great well, a second performs so-so because it is in a mediocre location but they designed the well very efficiently, and a third is doing poorly because they chose a bad location and did not design the well efficiently. In a program like Python one could calculate the Shapley values, which provide this granular information, but JMP is not able to do this. A fellow SE and I developed a method to calculate what I’ll call pseudo-Shapley but this procedure—while works very well—is only applicable to linear models and lacks the ability to break down results on a single well basis (row-by-row). In the O&G industry this capability would be ubiquitously deployable and open up JMP for expansion in this industry.

Re: Shapley values for predictive models - Status changed to: In Development

Currently in development and available in the JMP 17 EA.    

Re: Shapley values for predictive models - Status changed to: Yes, Stay Tuned!

There was significant work in this area for JMP 17, which will ship in October 2022. Marking as "Yes, Stay Tuned" with the expectation that someone will mark this as "Delivered" when the ship date comes along.