how to interpret the variable importance vs. profilerI have a customer doing a JMP Pro evaluation and they came up with a good question. Looking at the Diabetes data set, they ran a neural network on the response Y as a function of the factors: LTG, BMI, BP, HDL, Age, Glucose, LDL, TCH, Total Cholesterol, and Gender. The profiler shows that the effect of gender is small compared to the other factors:
From the profiler, the effect of varying Gender on Y is smaller than, say, BMI.
However, looking at the variable importance results, we see that Gender is at the top of the list:
Looking at the scatterplot matrix and the color map on correlations, using the dependent resampled inputs makes the most sense since the factors are correlated (as suggested in the Help).
The question from the customer is this: Why does the profiler show Gender having a small effect vs. the Variable Importance result which shows Gender having the largest effect?
The customer also asked what sample size was needed for the use of the Dependent Resampled Inputs results to be valid. The Help mentions that it doesn't work if the sample size is "too small" but doesn't say how small is small.
Checking the Statistical details for the Variable Importance says that it uses the Variance of the expected value of the response but doesn't give much detail. I assume that the difference between the profiler result and the Variable Importance results has something to do with the profiler using the expected value of Y and the Variable Importance result using the variance of the expected value of Y. However, I'd like more detail so I can explain it to the customer.
thanks!
Charlie
Re: how to interpret the variable importance vs. profilerWell, now I'm confused, as well.
I wondered if what you're seeing is due to the multicollinearity of all the factors. I see Gender as the least important variable when using both the Bootstrap Forest and Predictor Screening.
But, when I use Variable Clustering to reduce the predictors down to TCH, Total Cholesterol and BP, and also include Gender, then I see the same results as you do on variable importance:
So, now I'm questioning my understanding of the "Variable Importance" section, and I'm looking forward to the answer to your question.
Re: how to interpret the variable importance vs. profilerThanks Jed! As I mention, it may have something to do with the profiler using the expect value and the variable importance using the variance. But I'd like an expert to comment. Thanks!
Re: how to interpret the variable importance vs. profilerThe experts in our group on this subject are Ryan Parker and Chris Gotwalt. See their original work here https://community.jmp.com/t5/Discovery-Summit-2013/Exploring-Variable-Clustering-and-Importance-in-JMP/ta-p/40686
Here is my interpretation why the result appears to be so on this data, with this model and Dependent Resampled Inputs choice.
The customer's question is right on the money: the sample size!
When Dependent Resampled Inputs is selected, the fitted model has an intercept, 10 main factors, and 45 interaction terms. See page 10 in that Powerpoint file by Ryan. And Gender is categorical, additional columns are created in the design matrix. The model is really big. And 442 rows do not seem to be enough.
Re: how to interpret the variable importance vs. profilerYes, I think Peng is correct here. We had some internal discussion with Ryan and Chris and it's not completely obvious what's going on here, but Ryan's biggest concern hit on the fact that the sample size probably isn't large enough for the Dependent Resampled case.
Another explanation has to do with correlation between the inputs. Per Ryan, "For example, looking at the traces, you would think that LTG and HDL would have the highest impact. But looking at their correlation, they are negatively correlated and is likely one way to explain why in the dependent case the impact of those variables would be lower."
One thing I observed is that the use of Independent Resampled Inputs seems to give a more faithful picture of the relative size of the effects (similar to the Profiler and to other methods like Column Contributions on the BF model) but it's not suggested since we could be simulating at factor level combinations that are not representative of what we would actually see in real life.
Re: how to interpret the variable importance vs. profilerAh, but the power of JMP to explore outcomes that are not as we expected. If you turn on extrapolation control in the profilers and "play" with the factor settings, you start to see "holes" in the data. That is areas where the coverage for Males is different than females. This is biological data with blood values, and blood values can have different distributions due to gender (use fit y-by-x with gender as the X and the blood values as Ys and then use compare densities). So, there are subtle differences due to gender...and combinations in the data that are only in one gender (high HDL for example) which will impact resampling. It is the power to explore on the fly and to dig into data quickly that makes modeling in JMP fun!
Re: how to interpret the variable importance vs. profilerAdding on what @karencopeland mentioned, I just had sucha "Blood plasma" situation with a customer. They didn't have enough of one plasma and worked further with another (13 of o the first, 1 of the second and 3 more of a third plasma. Then they modeled it with plasma as blocking factor in the model. The results where ... uhm ... funny, especially if you turned on extrapolation control 
In addition the variable importance report can also differ quite a bit if you use a different option. So highly recomended to take a look at the discovery presentation
Re: how to interpret the variable importance vs. profilerThanks Martin! Good points!
Re: how to interpret the variable importance vs. profilerThanks Karen! I will have to look into the extrapolation control in the profilers to see.
