Hello, I'm Ron Kenneth.
This is a joint talk with Chris Gotwalt.
We said going to talk to you about models.
Models are used extensively.
We Hope to bring some additional perspective
on how to use models in general.
We call the talk
"Different goals, different models:
How to use models to sharpen your questions."
My part will be an intro, and I'll give an example.
You'll have access to the data I'm using.
Then Chris will continue with a more complex example,
introduce the SHAP values available in JMP 17,
and provide some conclusions.
We all know that all models are wrong,
but some are useful.
Sam Carlin said something different.
Hey said that the purpose of models is not to fit the data,
but to sharpen the question.
Then this guy, Pablo Picasso, hey said something in...
Hey died in 1973, so you can imagine when this was said.
i think in the early '70 s.
"Computers are useless. They can only give you answers."
Hey is more in line with Carlin.
My take on this is that
this presents the key difference between a model and a computer program.
I'm looking at the model from a statistician's perspective.
Dealing with Box's famous statement.
"Yes, some are useful.Which ones?"
Please help us.
What do you mean by some are useful?
It's not very useful to say that .
Going to Karlin, "Sharpen the question."
Okay, that's it a good idea.
How do well do that?
The point is that Box seems focused on the data analysis phase
in the life cycle view of statistics, which starts with problem elicitation,
moves to goal formulation, data collection, data analysis,
findings, operationalization of finding
communicational findings, and impact assessment.
Karlin is more focused on the problem elicitation phase.
These two quotes of Box and Carlin
refer to different stages in this life cycle.
The data I'm going to use is an industrial data set,
174 observations.
We have sensors data.
We have 63 sensors.
They are labelled 1, 2, 3, 4 to 63.
We have two response variables.
These are coming from testing some systems.
The status report is fail/pass.
52.8% of the systems that take tested failed.
We have another report,
which is a more detailed report on test results.
When the systems fail,
well have some classification of the failure.
Test result is more detailed.
Status is go/no go.
The goal is to determine the system status from the sensors data
so that well can maybe avoid the costs and delays of testing,
and well can have some early predictions on the faith of the system.
One approach well can take is to use a boosted tree.
We put the status as the response, the 63 sensors X, factors.
The boosted tree is trained sequentially, one tree at a time.
T hey other model then said going to use is random forest,
and that's it done with independent trees.
There is a sequential aspect in boosted trees
that is different from random forests.
The setup of boosted trees involves three parameters:
the number of trees, depth of trees, and the learning rate.
This is what JMP gives as a default.
Boosted trees can be used to solve most objective functions.
We could use it for poisson regression,
which is dealing with counts that is a bits harder to achieve
with random forests.
We said going to focus on these two types of models.
When well apply the boosted tree,
and well have a validation set up
with 43 systems drawn randomly as the validation set.
A hundred and thirty-one systems is used for the training set.
We are getting a 9.3% misclassification rate.
Three failed systems.
We know that they failed because well have it in the data,
take actually classified as pass.
The 20 that passed, 19 take classified as pass.
The false predicted pass is 13 %.
We can look at the variable column contributions.
We see that Sensor 56, 18, 11, and 61 are the top four
in terms of contributing to this classification.
We see that in the training set, well had zero misclassification.
We might have some over fitting i n this BT application.
If well look at the lift curve,
40 % of the systems, well can get over two lift
which is the performance that this classifier gives us.
If well try the boos trap forest,
another options, again, well do the same thing.
We use the same validation set.
The defaults of JMP are giving you some parameters
for the number of trees
and the number of features to be selected at each mode.
This is how the random forest works.
You should be aware that this is not very good
if you have categorical variables and missing data,
which is not our case here.
Now, the misclassification rate is 6.9, lower than before.
Mr the training set, well had some misclassification.
The random forest applied to the test status,
which means when well have the details on the failures is 23.4,
so bad performance.
Also, on the training set, well have 5% misclassification.
But well have now a wider range of options
and that is explaining some of the lower performance.
In the lift curve on the test results,
well actually with quite good performance,
can pick up the top 10 % of the systems with a leverage of above 10.
So well have over a ten fold increase for 10 % of the systems
relative to the grand average.
Now this is posing a questionâ remember the topic of the talkâ
what are well looking at?
Do well want to identify top score good systems?
The random forest would do that with the test result.
Or do well want to predict a high proportion of pass?
The bootstrap tree would offer that.
A secondary question is to look at what is affecting this classification.
We can look at the column contributions on the boosted tree.
Three of the four top variables show up also on the random forest.
If well use the status pass/fail,
or the detailed results,
there is a lots of similarity on the importance of the sensors.
This is just giving you some background.
Chris is going to follow up with an evaluation of the sensitivity
of this variable importance, the use of SHAP v values
and more interesting stuff.
This goes back to questioning what is your goal,
and how is the model helping you figure out the goal
and maybe sharpening the question that comes from the statement of the goal.
Chris, it's all yours.
Thanks, Ron.
I'm going to pick up from where Ron left off
and seek a model that will predict whether or not a unit is good or not,
and if it isn't what's the probably failure mode that has result?
This would be useful in that if a model is good at predicting good units,
well May not have to subject them to much further testing.
If the model gives a predicted failure mode,
then said able to get a head start on diagnosing and fixing the problems,
and possibly, well May be able to get some hints
on how to improve the production process in the future.
I'm going to go through the sequence
of how i approached answering this question from the data.
i want to say at the output that this is simply the path that i take
as i asked questions of the data and acted on various patterns that i see.
There are literally many other ways that one could proceed with this.
There's often not really a truly correct answer,
just a criterion for whether or not the model is good enough
and the amount that you're able to get done
in the time that you have to get a result back.
i have no doubt that there are better models out there
than what i came up with here.
Our goal is to show an actual process of tackling a prediction problems,
illustrating how one can move forward
by iterating through cycles of modeling and visualization,
followed by observing the results and using them to ask another question
until well find something of an answer.
i will be using JMP as a statistics Switzerland army knife,
using many tools in JMP
and following the intuition i have about modeling data
that has built up over many years.
First, let's just take a look
at the frequencies of the various test result categories.
We see that the largest and most frequent category is Good.
We'll probably have the most luck being able to predict that category.
Mr the other hands, the SOS category has only two events
so it's going to be very difficult for us to be able to do much with that category.
We May have to set that one apart.
We'll see about that.
Velocity II, IMP, and Brake
are all fairly rare with five or six events each.
There May be some limitations in what then said able to do with them as well.
i say this because well have 174 observations
and well have 63 predictors.
So well have a lots of predictors for a very small number of observations,
which is actually even smaller when you consider the frequencies
of some of the categories that then said trying to predict.
We said going to have to work iteratively by doing visualizations in modelling,
recognizing patterns, asking questions,
and then acting on those with another modeling step iteratively
in order to find a model that's it going to do a good job
of predicting these response categories.
i have the data sorted by test results,
so that the good results are at the beginning,
followed by each of the different failure modes d this bite fter that.
i went ahead and colored each of the rows by test results so that well can see
which observation belongs to a particularly response category.
So then i went into the model- driven multivariate control chart
and i brought in all of the sensors as process variables.
Since i had the good test results at the beginning,
i labelled those as historical observations.
This gives us a T² chart.
It's chosen 13 principal components as its basis.
What well see here
is that the chart is dominated by these two points right here
and all of the other points are very small in value
relative to those two.
Those two points happen to be the SOS points.
They are very serious outliers in the sensors readings.
Since well also only have two observations of those,
I'm going to go ahead and take those out of the data set
and say well, SOS is obviously so bad that the sensors
should be just flying off the charts.
If well encounter it, then said just going to go ahead
and try to concern ourselves with the other values
that don't have this off- the- charts behavior.
Switching to a log scale, well see that the good test results
are fairly well -behaved.
Then there's definitely signals
in the data for the different failure modes.
Now well can drill down a little bits deeper,
taking a look at the contribution plots for the historical data,
the good test result data, and the failure modes
to see if us patterns emerge in the sensors that well can act upon.
I'm going to remove those two SOS observations
and select the good units.
If i right-click in the plot,
i can bring up a contribution plot
for the good units, and then i can go over to the units
where there was a failure, and i can do the same thing
and we'll be able to compare the contribution plots side by side.
So what well see here are the contribution plots
for the pass units and the fail units.
The contribution plots
are the amount that each column is contributing to the T ²
for a particularly row.
Each of the bars there correspond to an individual sensors for that row.
Contribution plots are colored green when that column is within three sigma,
using an individuals and moving range chart,
and it's red if it's out of control.
Look well see most of the sensors are in control for the good units,
and most of the sensors are out of control for the failed units.
What i was hoping for here
would have been if there was only a subset of the columns
or sensors that take out of control over on the failed units.
Or if i was able to see patterns
that changed across the different failure modes,
which would help me isolate what variables are important
for predicting the test result outcome.
Unfortunately, pretty much all of the sensors
are in control when things are good,
and most of them are out of control when things are bad.
So we're going to have to use some more sophisticated modeling
to be able to tackle this prediction problem.
Having not found anything in the column contributions plots,
I'm going to back up and return to the two models that Ron found.
Look are the column contributions for those two models,
and well see that there's some agreement in terms of
what are the most important sensors.
But boosted tree found a somewhat larger set of sensors as being important
over the bootstrap forest.
Which of these two models should well trust more?
If well look at the overall model fit report,
well see that the boosted tree model has a very high training Rs quare of 0.998
and a somewhat smaller v alidation Rs quare of 0.58.
This looks like an overfit situation.
When well look at the random forest, it has a somewhat smaller training Rs quare,
perhaps a more realistic one, than the bootstrap forest,
and it has a somewhat larger validation Rs quare.
The generalization performance of the random forest
is hopefully a little bits better.
I'm inclined to trust the random forest model a little bits more.
Part of this is going to be based upon just the folklore of these two models.
Boosted trees are renowned for being fast, highly accurate models
that work well on very large datasets.
Whereas the hearsay is that random forests are more accurate on smaller datasets.
They are fairly robust, messy, and noisy data.
There's a long history of using these kind of models
for variable selection that goes back to a paper in 2010
that has been cited almost 1200 times.
So this is a popular approach for variable selection.
i did a similar search for boosting,
and i didn't quite see as much history around variable selection
for boosted trees as i did for random forests.
For this given data set r right here,
well can do a sensitivity analysis to see how reliable
the column contributions are for these two different approaches,
using the simulation capabilities in JMP Prof.
What well can do is create a random validation column
that is a formula column
that you can reinitialize and will partition the data
into random training and holdout sets of the same portions
as the original validation column.
We can have it rerun these two analyses
and keep track of the column contribution portions
for each of these repartitionings.
We can see how consistent the story is
between the boosted tree models and the random forests.
This is pretty easy to do.
We just go to the Make Validation Column utility
and when well make a new column, well ask it to make a formula column
so that it could be reinitialized.
Then well can return to the bootstrap forest platform,
right- click on the column contribution portions,
select Simulate.
It'll bring up a dialog
asking us which of the input columns well want to switch out.
I'm going to choose the validation column,
and I'm going to switch in in replacement of it,
this random validation formula column.
We said going to do this a hundred times.
Bootstrap forest is going to be rerun
using new random partitions of the data into training and validation.
We said going to look at the distribution of the portions
across all the simulation runs.
This will generate a dataset
of column contribution portions for each sensors.
We can take this and go into the graph builder
and take a look and see how consistent those column contributions are
across all these random partitions of the data.
Look is a plot of the column contribution portions
from each of the 100 random reshufflings of the validation column.
Those points well see in gray here,
Sensor 18 seems to be consistently a big contributor, as does Sensor 61.
We also see with these red crosses,
those are the column contributions from the original analysis that Ron did.
The overall story that this tells is that the tendency
i s that whenever the original column contribution was small
those re simulated column contributions also tended to be small.
When the column contributions take large in the original analysis,
they tended to stay large.
We said getting a relatively consistent story from the bootstrap forest
in terms of what columns are important.
Now well can do the same thing with the boosted tree,
and the results aren't quite as consistent as they take with the bootstrap forest.
So here is a bunch of columns
where the initial column contributions came out very small
but they had a more substantial contribution
in some of the random reshuffles of the validation column.
That also happened quite a bits over with these Columns 52 through 55 over here.
Then there take also some situations
where the original column contribution was quite big,
and most, if not all,
of the other column contributions found in the simulations take smaller.
That happens here with Column 48,
and to some extent also with Column 11 over here.
The overall conclusion being that i think this validation shuffling
is indicating that well can trust the column contributions
from the bootstrap forest to be more stable than those of the boosted tree.
Based on this comparison, i think i trust the column contributions
from the bootstrap forest more,
and I'm going to use the columns that it recommended
as the basis for some other models.
What I'd like to do
is find a model that is both simpler than the bootstrap forest model
and performs better in terms of validation set performance
for predicting pass or fail.
Before proceeding with the next modeling stuff
I'm going to do something that i should have probably done at the very beginning,
which is to take a look at the sensors in a scatterplot matrix
to see how correlated the sensors readings are,
and also look at histograms of them as well to see if they're outlier prone
or heavily skewed or otherwise highly non- gaussian.
What well see here is there is pretty strong multicollinearity
amongst the input variables generally.
We said only looking at a subset of them here,
but this high multicollinearity persists across all of the sensors readings.
This suggestions that for our models,
well should try things like the logistics lasso,
the logistics elastic internet, or a logistics ridge regression
as candidates for our model to predict pass or fail.
Before well do that well should go ahead
and try to transform our sensors readings here
so that they're a little bits better- behaved and more gaussian- looking.
This is actually really easy in JMP
if you have all of the columns up in the distribution platform,
because all you have to do is hold down Alt , choose Fit Johnson,
and this will fit Johnson distributions to all the input variables.
This is a family of distributions
that is based on a four parameters transformation to normality.
As a result, well have a nice features in there
that well can also broadcast using Alt Click,
where well can save a transformation from the original scale
to a scale that makes the columns more normally distributed.
If well go back to the data table,
we'll see that for each sensors column, a transform column has been added.
If well bring these transformed columns up with a scatterplot matrix
and some histograms,
well obviously see that the data are less skewed
and more normally distributed than the original sensors columns take.
Now the bootstrap forest model that Ron found
only really recommended a small number of columns
for use in the model.
Because of the high collinearity amongst the columns,
the subset that well got could easily be part
of a larger group of columns that are correlated with one another.
It could be beneficial to find that larger group of columns
and work with that at the next modeling stage.
An exploratory way to do this
is to go through the cluster variables platform in JMP .
We said going to work with the normalized version of the sensors
because this platform is PCA and factor analysis based,
and will provide more reliable results if then said working with data
that are approximately normally distributed.
Once then said in the variable clustering platform,
well see that there is very clear,
strong associations amongst the input columns.
It has identified that there are seven clusters,
and the largest cluster, the one that explains the most variations,
has 25 members.
The set of cluster members is listed here on the right.
Let's compare this with the bootstrap forest.
Look on the left, well have the column contributions
from the bootstrap forest model that you should be familiar with by now.
Mr the right well have the list of members
of that largest cluster of variables.
If well look closely we'll see that the top seven contributing terms
all happen to belong to this cluster.
I'm going to hypothesize that this set of 25 columns
are all related to some underlying mechanism
that causes the units to pass or fail.
What i want to do next is i want to fit models
using the generalized regression platform with the variables in Cluster 1 here.
It would be tedious if i had to go through
and individually pick these columns out and put them into the launch dialog.
Fortunately, there's a much easier way
where you can just select the rows in that table
and the columns will be selected in the original data table
so that when well go into the fit model launch dialog,
well can just click Add
and those columns will be automatically added for us as model effects.
Once i got into the Generalized Regression platform,
i went ahead and fit a lasso model and elastic net model
and a ridge model to have them compared here to each other,
and also to the logistics regression model that came up by default.
We said seeing that the lasso model is doing a little bits better than the rest
in terms of its validation generalized Rs quare.
The difference between these methods
is that there's different amounts of variable selection
and multicollinearity handling in each of them.
Logistic regression has no multicollinearity handling
and no variable selection.
The lasso is more of a variable selection algorithm,
though it has a little bits of multicollinearity handling in it
because it's a penalized method.
Ridge regression has no variable selection
and is heavily oriented around multicollinearity handling.
The elastic net is a hybrid between the lasso and ridge regression.
In this case, what well really care about
is just the model that's it going to perform the best.
We allow the validation to guide us.
We said going to be working with the lasso model from here on.
Here's the prediction profiler for the lasso model that was selected.
We see that the lasso algorithm has selected eight sensors
as being predictive of pass or fail.
It has some great built-in tools
for understanding what the important variables are,
both in the model overall and new to JMP Prof 17,
well have the ability to understand
what variables are most important for an individual prediction.
We can use the variable importance tools to answer the questions,
"What are the important variables in the model?"
We have a variety of different options for this.
We have a variety of different options for how this could be done.
But because of the multicollinearity and because this is not a very large models,
I'm going to go ahead
and use the dependent resampled inputs technique,
since well have multicollinearity in the data,
and this has given us a ranking of the most important terms.
We see that Column 18 is the most important,
followed by Column 27 and then 52, all the way down.
We can compare this to the bootstrap forest models,
and well see that there's agreement that Variable 18 is important,
along with 52, 61, and 53.
But one of the terms that well have pulled in
because of the variable clustering step that well had done,
Sensor 27 turns out to be the second most important predictive
in this lasso model.
We've hopefully gained something by casting a wider net through that step.
We've found a term that didn't turn up
in either of the bootstrap forest or the boosted tree methods.
We also see that the lasso model has an Rs quare of 0.9,
whereas the bootstrap forest model had an Rs quare of 0.8.
We have a simpler model that has an easier form to understand
and is easier to work with
and also has a higher predictive capacity than the bootstrap forest model.
Now, the variable importance metrics in the profiler
have been there for quite some time.
The question that they answer is "Which predictors have the biggest impact
on the shape of the response surface over the data or over a region?"
In JMP 17 pro, well have a new technique called SHAP Values
that is an additive decomposition of an individual prediction.
It tells you by how much each individual variable contributes
to a single prediction,
rather than talking about variability explained over the whole space.
The resolution of the question that's it answered by Shapley values
is far more local than either the variable importance tools
or the column contributions i n the bootstrap forest.
We can obtain the Shapley Values by going to the red triangle menu for the profiler,
and we'll find the option for them over here, fourth from the bottom.
When well choose the options, the profiler saves back SHAP columns
for all of the input variables to the model.
This is of course, happening for every row in the table.
What you can see is that the SHAP V values are giving you the effect
of each of the columns on the predictive model.
This is useful in a whole lots of different ways,
and for that reason, it's got a lots of attention in intelligible AI,
because it allows us to see
what the contributions are of each column to a black box model.
Here, I've plotted the SHAP V values for the columns that are predictive
in the last model fit that well just built.
If well toggle back and forth between the good units and the units that failed,
well see the same story that we've been seeing
with the variable importance metrics for this
that Column 18 and Column 27 are important in predicting pass or fail.
We said seeing this at a higher level of resolution
than well do with the variable importance metrics,
because each of these points corresponds to an individual row
in the original dataset.
But in this case, i don't see the SHAP Values
really giving us us new information.
i had hoped that by toggling through
the other failure modes, maybe i could find a pattern
to help tease out different sensors
that are more important for particularly failure modes.
But the only thing i was able to find was that Column 18
had a somewhat stronger impact on the Velocity Type 1 failure mode
than the other failure modes.
At this point, we've had some successful
using those Cluster 1 columns in a binary pass/ fail model.
But when i broken out the SHAP Values
for that models, by the different failure modes
i wasn't able to discern a pattern or much of a pattern.
What i did next was i went ahead
and fit the failure mode response column test results
using the Cluster 1 columns,
but i went ahead and excluded all the pass rows
so that the modeling procedure would focus exclusively
on discerning which failure mode it is given that well have a failure.
i tried the multinomial lasso, elastic internet, and ridge,
and i was particularly happy with the lasso model
because it gave me a validation Rs quare of about 0.94.
Having been pretty happy with that
i went ahead and saved the probability formulas
for each of the failure modes.
Now the task is to come up with a simple rules
that post processes that prediction formula
to make a decision about which failure mode.
i call this the partition trick.
The partition trick is where i put in the probability formulas
for a categorical response, or even a multinomial response.
i put those probability formulas in as Xs.
i use my categorical response as my Y.
This is the same response that was used
for all of these except for pass actually.
i retain the same validation column that I've been working with the whole time.
Now that I'm in the partition platform,
I'm going to hit Split a couple of times, and I'm going to Hope
that i end up with an easily understandable decision rules
that's it easy to communicate.
That May or May not happen.
Sometimes it works, sometimes it doesn't.
So i split once, and well end up seeing that
whenever the probability of pass is higher than 0.935,
well almost certainly have a pass.
Not many passes are left over on the other side.
i take another split.
We find a decision rules on ITM
that is highly predictive of ITM as a failure mode.
Split again.
We find that whenever Motor is less than 0.945,
then said either predicting Motor or Brake.
We take another split.
We find that whenever Velocity Type 1, its probability is bigger than 0.08
or likely in a Velocity Type 1 situation or in a Velocity T ype 2 situation.
Whenever Velocity Type 1 is less than 0.79,
then said probably in a gripper failure mode or an IMP failure mode.
What do well have here? We have a simple decision rules.
We said going to not be able to break these failure modes down much further
because of the very small number of actual events that well have.
But well can turn this into a simple rules
for identifying units that are probably good,
and if they're not, well have an idea of where to look to fix the problem.
We can save this decision rules out as a leaf label formula.
We see that on the validation set,
when well predict it's good, it's good most of the time.
We did have one misclassification of a Velocity Type 2 failure
that was actually predicted to be good.
Predict grippers or IMP, it's all over the place.
That leaf was not super useful.
Predicting ITM is 100 %.
Whenever well predict a motor or brakes,
on the validation set, well have a motor or a brake failure.
When well predict a Velocity Type 1 or 2,
it did a pretty good job of picking that up
with that one exception of the single Velocity Type 2 unit
that was in the validation set,
and that one happened to have been misclassified.
We have an easily operational rules here that could be used to sort products
and give us a head start on where well need to look to fix things.
i think this was a pretty challenging problems,
because well didn't have a whole lots of data.
But well didn't have a lots of rows,
but well had a lots of different categories to predict
and a whole lots of possible predictors to use.
We've got there by taking a series of steps,
asking questions,
sometimes taking a step back and asking a bigger question.
Other times, narrowing in on particularly sub- issues.
Sometimes our excursions take fruitful, and sometimes they weren't.
Our purpose here is to illustrate
how you can step through a modeling process,
through this sequence of asking questions
using modeling and visualization tools to guide your next steps,
and moving on until you're able to find
a useful, actionable, predictive model.
Thank you you very much for your attention.
We look forward to talking to you in our Q&A session coming up next.
