Hello,
I'm
Ron
Kennett.
This
is
a
joint
talk
with
Chris
Gotwalt.
We're
going
to
talk
to
you
about
models.
Models are
used
extensively.
We
hope
to
bring
some
additional
perspective
on
how
to
use
models
in
general.
We
call
the
talk
"Different
goals,
different
models:
How
to
use
models
to
sharpen
your
questions."
My
part
will
be
an
intro,
and
I'll
give
an
example.
You'll
have
access
to
the
data
I'm
using.
Then
Chris
will
continue
with a
more
complex
example,
introduce
the
SHAP
values
available
in
JMP
17,
and
provide
some
conclusions.
We
all
know
that
all
models
are
wrong,
but
some
are
useful.
Sam
Karlin
said
something
different.
He
said
that
the
purpose
of
models
is
not
to
fit
the
data,
but
to
sharpen
the
question.
Then
this
guy,
Pablo
Picasso,
he
said
something in...
He
died
in
1973,
so
you
can
imagine
when
this
was
said.
I
think
in
the
early
'70
s.
"Computers
are
useless.
They
can
only
give
you
answers."
He
is
more
in
line
with
Karlin.
My
take
on
this is that
this presents
the key difference
between
a
model
and
a
computer
program.
I'm
looking
at
the
model
from
a
statistician's
perspective.
Dealing
with
Box's
famous
statement.
"Yes,
some
are
useful.
Which
ones?"
Please
help
us.
What do
you
mean
by
some
are
useful?
It's
not
very
useful
to
say
that
.
Going to Karlin,
"Sharpen
the
question."
Okay,
that's
a
good
idea.
How
do
we
do
that?
The
point
is
that
Box
seems
focused
on the data analysis phase
in the life cycle view of statistics,
which starts with problem elicitation,
moves
to
goal
formulation,
data
collection,
data
analysis,
findings,
operationalization
of
finding,
communicational
findings,
and
impact
assessment.
Karlin is
more
focused
on
the
problem
elicitation
phase.
These
two
quotes
of
Box
and
Karlin
refer
to
different
stages
in
this
life
cycle.
The
data
I'm
going
to
use
is
an
industrial
data
set,
174 observations.
We
have
sensor
data.
We
have
63
sensors.
They are
labeled
1, 2, 3, 4
to
63.
We
have
two
response
variables.
These
are
coming
from
testing
some
systems.
The
status
report
is
fail/pass.
52.8%
of
the
systems
that
were
tested
failed.
We
have
another
report,
which
is
a
more
detailed
report
on
test
results.
When
the systems
fail,
we
have
some
classification
of
the
failure.
Test
result
is
more
detailed.
Status
is
go/no go.
The
goal
is
to
determine
the
system
status
from
the
sensor
data
so
that
we
can
maybe
avoid
the
costs
and
delays
of
testing,
and
we
can
have
some
early
predictions
on
the
faith
of
the
system.
One
approach
we
can
take
is
to
use
a
boosted
tree.
We
put
the
status
as the response, the 63 sensors X, factors.
The
boosted
tree
is
trained
sequentially,
one
tree
at
a
time.
T
he
other
model
we're
going
to
use
is
random
forest,
and
that's
done
with
independent
trees.
There
is
a
sequential
aspect
in
boosted
trees
that
is
different
from
random
forests.
The
setup
of
boosted
trees
involves
three
parameters:
the
number
of
trees,
depth
of
trees,
and
the
learning
rate.
This
is
what
JMP
gives
as
a default.
Boosted
trees
can
be
used
to
solve
most
objective
functions.
We
could
use
it
for
poisson
regression,
which
is
dealing
with
counts
that
is
a
bit
harder
to
achieve
with
random
forests.
We're
going
to
focus
on
these
two
types
of
models.
When
we
apply
the
boosted
tree,
and
we
have
a
validation
set
up
with
43
systems
drawn
randomly
as
the
validation
set.
A hundred and thirty-one systems
is used
for
the
training
set.
We
are
getting
a
9.3%
misclassification
rate.
Three
failed
systems.
We
know
that they
failed
because
we
have
it
in
the
data,
were
actually
classified
as
pass.
The
20
that
passed,
19
were
classified
as
pass.
The
false
predicted
pass
is
13
%.
We
can
look
at
the
variable
column
contributions.
We
see
that
Sensor
56,
18,
11,
and
61
are
the
top
four
in
terms
of
contributing
to
this
classification.
We
see
that
in
the
training
set,
we
had
zero
misclassification.
We
might
have
some
over
fitting i
n
this
BT
application.
If
we
look
at
the
lift
curve,
40
%
of
the
systems,
we
can
get
over
two
lift
which
is
the performance that this classifier gives us.
If
we
try
the
boos
trap
forest,
another
option,
again,
we
do
the
same
thing.
We
use
the
same
validation
set.
The
defaults
of
JMP
are
giving
you some
parameters
for
the
number
of
trees
and
the
number
of
features
to
be
selected
at
each
mode.
This
is
how
the
random
forest
works.
You
should
be
aware
that
this
is
not
very
good
if
you
have
categorical
variables
and
missing
data,
which
is
not
our
case
here.
Now,
the
misclassification
rate
is
6.9,
lower
than
before.
On
the
training
set,
we
had
some
misclassification.
The
random
forest
applied
to
the
test
status,
which
means
when
we
have
the
details
on
the
failures
is
23.4,
so
bad
performance.
Also,
on
the
training
set,
we
have
5%
misclassification.
But
we
have
now
a
wider
range
of
options
and that is explaining
some
of
the
lower
performance.
In
the
lift
curve
on
the
test
results,
we
actually,
with
quite
good
performance,
can
pick
up
the
top
10
%
of
the
systems
with
a
leverage
of
above
10.
So
we
have
over a ten
fold
increase
for
10
%
of
the
systems
relative
to
the
grand
average.
Now
this
is
posing
a
questionâ
remember
the
topic
of
the
talkâ
what
are
we
looking
at?
Do
we
want
to
identify
top
score
good
systems?
The
random
forest
would
do
that
with
the
test
result.
Or
do
we
want
to
predict
a
high
proportion
of
pass?
The
bootstrap tree
would
offer
that.
A
secondary
question
is
to
look
at
what
is
affecting
this
classification.
We
can
look
at
the
column
contributions
on
the
boosted
tree.
Three
of
the
four top
variables
show
up
also
on
the
random
forest.
If
we
use
the
status
pass/fail,
or
the
detailed
results,
there
is
a
lot
of
similarity
on
the
importance
of
the
sensors.
This
is
just
giving
you
some
background.
Chris
is
going
to
follow
up
with
an
evaluation
of
the
sensitivity
of
this
variable
importance,
the
use
of
SHAP v
alues
and more interesting stuff.
This
goes
back
to
questioning
what
is
your
goal,
and
how
is
the
model
helping
you
figure
out
the
goal
and
maybe
sharpening
the
question
that
comes
from
the
statement
of
the
goal.
Chris,
it's
all yours.
Thanks,
Ron.
I'm
going
to
pick
up
from
where
Ron
left off
and
seek
a
model
that
will
predict
whether
or
not
a
unit
is
good
or
not,
and
if
it
isn't,
what's
the
likely
failure
mode
that
has
resulted?
This
would
be
useful
in that
if
a
model
is
good
at
predicting
good
units,
we
may
not
have
to
subject
them
to
much
further
testing.
If
the
model
gives
a
predicted
failure
mode,
we're
able
to
get
a
head
start
on
diagnosing
and
fixing
the
problem,
and
possibly,
we
may
be
able
to
get
some
hints
on
how
to
improve
the
production
process
in
the
future.
I'm
going
to
go
through
the
sequence
of
how
I
approached
answering
this
question
from
the
data.
I
want
to
say
at
the
outset
that
this
is
simply
the
path
that
I
took
as
I
asked
questions
of
the
data
and
acted
on
various
patterns
that
I
saw.
There
are
literally
many
other
ways
that
one
could
proceed
with
this.
There's
often
not
really
a
truly
correct
answer,
just
a
criterion
for
whether
or
not
the
model
is
good
enough,
and
the
amount
that
you're
able
to
get
done
in
the
time
that
you
have
to
get
a
result
back.
I
have
no
doubt
that
there
are
better
models
out
there
than
what
I
came
up
with
here.
Our
goal
is
to
show
an
actual
process
of
tackling
a
prediction
problem,
illustrating
how
one
can
move
forward
by
iterating
through
cycles
of
modeling
and
visualization,
followed
by
observing
the
results
and
using
them
to
ask
another
question
until
we
find
something
of
an
answer.
I
will
be
using
JMP
as
a
statistical
Swiss
army
knife,
using
many
tools
in
JMP
and
following
the
intuition
I
have
about
modeling
data
that
has
built
up
over
many
years.
First,
let's
just
take
a
look
at
the
frequencies
of
the
various
test
result
categories.
We
see
that
the
largest
and
most
frequent
category
is
Good.
We'll
probably
have
the
most
luck
being
able
to
predict
that
category.
On
the
other
hand,
the
SOS
category
has
only
two
events
so
it's
going
to
be
very
difficult
for
us
to
be
able
to
do
much
with
that
category.
We
may
have
to
set
that
one
aside.
We'll
see
about
that.
Velocity II,
IMP,
and
Brake
are
all
fairly
rare
with
five
or
six
events
each.
There
may
be
some
limitations
in
what
we're
able
to
do
with
them
as
well.
I
say
this
because
we
have
174
observations
and
we
have
63
predictors.
So
we
have
a
lot
of
predictors
for
a
very
small
number
of
observations,
which
is
actually
even
smaller
when
you
consider
the
frequencies
of
some
of
the
categories
that
we're
trying
to
predict.
We're
going
to
have
to
work
iteratively
by
doing
visualizations
in
modeling,
recognizing
patterns,
asking
questions,
and
then
acting
on
those
with
another
modeling
step
iteratively
in
order
to
find
a
model
that's
going
to
do
a
good
job
of
predicting
these
response
categories.
I
have
the
data
sorted
by
test
results,
so
that
the
good
results
are
at
the
beginning,
followed
by
each
of
the
different
failure
modes d
ata a
fter
that.
I
went
ahead
and
colored
each
of
the
rows
by
test
results
so
that
we
can
see
which
observation
belongs
to
a
particular
response
category.
So
then
I
went
into
the
model-
driven
multivariate
control
chart
and
I
brought
in
all
of
the
sensors
as
process
variables.
Since
I
had
the
good
test
results
at
the
beginning,
I
labeled
those
as
historical
observations.
This
gives us
a
T²
chart.
It's
chosen
13
principal
components
as
its
basis.
What
we
see
here
is
that
the
chart
is
dominated
by
these
two
points
right
here
and
all
of
the
other
points
are
very
small
in
value
relative
to
those
two.
Those two points happen
to
be
the SOS
points.
They
are
very
serious
outliers
in
the
sensor
readings.
Since
we
also
only
have
two
observations
of
those,
I'm
going
to
go
ahead
and
take
those
out
of
the
data
set
and
say,
well, SOS is
obviously
so
bad
that
the
sensors
should
be
just
flying
off
the
charts.
If
we
encounter
it,
we're
just
going
to
go
ahead
and
try
to
concern
ourselves
with
the
other
values
that
don't
have
this
off-
the-
charts
behavior.
Switching
to
a
log
scale,
we
see
that
the
good
test
results
are
fairly
well
-behaved.
Then
there's
definite
signals
in
the
data
for
the
different
failure
modes.
Now
we
can
drill
down
a
little
bit
deeper,
taking
a
look
at
the
contribution
plots
for
the
historical
data,
the
good
test
result
data,
and
the
failure
modes
to
see
if
any
patterns
emerge
in
the
sensors
that
we
can
act
upon.
I'm
going
to
remove
those
two SOS
observations
and
select
the
good
units.
If
I
right-click
in
the
plot,
I
can
bring
up
a
contribution
plot
for
the
good
units,
and
then
I
can
go
over
to
the
units
where
there
was
a
failure,
and
I
can
do
the
same
thing,
and
we'll
be
able
to
compare
the
contribution
plots
side
by
side.
So
what
we
see
here
are
the
contribution
plots
for
the
pass
units
and
the
fail
units.
The
contribution
plots
are
the
amount
that
each
column
is
contributing
to
the
T
²
for
a
particular
row.
Each of
the
bars
there
correspond
to
an
individual
sensor
for
that
row.
Contribution
plots
are
colored
green
when
that
column
is
within
three
sigma,
using
an
individuals
and
moving
range
chart,
and
it's
red
if
it's
out
of
control.
Here
we
see
most
of
the
sensors
are
in
control
for
the
good
units,
and
most
of
the
sensors
are
out
of
control
for
the
failed
units.
What
I
was
hoping
for
here
would
have
been
if
there
was
only
a
subset
of
the
columns
or
sensors
that
were
out
of
control
over on
the
failed
units.
Or
if
I
was
able
to
see
patterns
that
changed
across
the
different
failure
modes,
which
would
help
me
isolate
what
variables
are
important
for
predicting
the
test
result
outcome.
Unfortunately,
pretty
much
all
of
the
sensors
are
in
control
when
things
are
good,
and
most
of
them
are
out
of
control
when
things
are
bad.
So we're going to
have
to
use
some
more
sophisticated
modeling
to
be
able
to
tackle
this
prediction
problem.
Having
not
found
anything
in
the
column
contributions
plots,
I'm
going
to
back
up
and
return
to
the
two
models
that
Ron
found.
Here
are
the
column
contributions
for
those
two
models,
and
we
see
that
there's
some
agreement
in
terms
of
what
are
the
most
important
sensors.
But
boosted
tree
found
a
somewhat
larger
set
of
sensors
as
being
important
over
the
bootstrap
forest.
Which
of
these
two
models
should
we
trust
more?
If
we
look
at
the
overall
model
fit
report,
we
see
that
the
boosted
tree
model
has
a
very
high
training
RS
quare
of
0.998
and
a
somewhat
smaller v
alidation
RS
quare
of
0.58.
This
looks
like
an
overfit
situation.
When
we
look
at
the
random
forest,
it
has
a
somewhat
smaller
training
RS
quare,
perhaps
a
more
realistic
one,
than
the
bootstrap
forest,
and
it has
a
somewhat
larger
validation
RS
quare.
The
generalization
performance
of
the
random
forest
is
hopefully
a
little
bit
better.
I'm
inclined
to
trust
the
random
forest
model
a
little
bit
more.
Part
of
this
is
going
to
be
based
upon
just
the
folklore
of
these
two
models.
Boosted
trees
are
renowned
for
being
fast,
highly
accurate
models
that
work
well
on
very
large
datasets.
Whereas
the
hearsay
is
that
random
forests
are
more
accurate
on
smaller
datasets.
They
are
fairly
robust,
messy,
and
noisy
data.
There's
a
long
history
of
using
these
kinds
of
models
for
variable
selection
that
goes
back
to
a
paper
in
2010
that
has
been
cited
almost
1200
times.
So
this
is
a
popular
approach
for
variable
selection.
I
did
a
similar
search
for
boosting,
and
I
didn't
quite
see
as
much
history
around
variable
selection
for
boosted
trees
as
I
did
for
random
forests.
For
this
given
data
set r
ight
here,
we
can
do
a
sensitivity
analysis
to
see
how
reliable
the
column
contributions
are
for
these
two
different
approaches,
using
the
simulation
capabilities
in
JMP
Pro.
What
we
can
do
is
create
a
random
validation
column
that
is
a
formula
column
that
you
can
reinitialize
and
will
partition
the
data
into
random
training
and
holdout
sets
of
the
same
portions
as
the
original
validation
column.
We
can
have
it
rerun
these
two
analyses
and
keep
track
of
the
column
contribution
portions
for
each
of
these
repartitionings.
We
can
see
how
consistent
the
story is
between
the
boosted
tree
models
and
the
random
forests.
This
is
pretty
easy
to
do.
We just go
to
the
Make
Validation
Column
utility
and
when
we
make
a
new
column,
we
ask
it
to
make
a
formula
column
so
that it
could
be
reinitialized.
Then
we
can
return
to
the
bootstrap
forest
platform,
right-
click
on
the
column
contribution
portion,
select
Simulate.
It'll
bring
up
a
dialog
asking
us
which
of
the
input
columns
we
want
to
switch
out.
I'm
going
to
choose
the
validation
column,
and
I'm
going
to
switch
in
in replacement
of
it,
this
random
validation
formula
column.
We're
going
to
do
this a hundred
times.
Bootstrap
forest
is
going
to
be
rerun
using
new
random
partitions
of
the
data
into
training
and
validation.
We're
going
to
look
at
the
distribution
of
the
portions
across
all
the
simulation
runs.
This
will
generate
a
dataset
of
column
contribution
portions
for
each
sensor.
We
can
take
this
and
go
into
the
graph
builder
and
take
a
look
and
see
how
consistent
those
column
contributions
are
across
all
these
random
partitions
of
the
data.
Here
is
a
plot
of
the
column
contribution
portions
from
each
of
the
100
random
reshufflings
of
the
validation
column.
Those points
we
see
in
gray here,
Sensor
18
seems
to
be
consistently
a
big
contributor,
as
does
Sensor
61.
We
also
see
with
these
red
crosses,
those
are
the
column
contributions
from
the
original
analysis
that
Ron
did.
The
overall
story
that
this
tells
is
that
the
tendency
i
s
that
whenever
the
original
column
contribution
was small,
those
re
simulated
column
contributions
also
tended
to
be
small.
When
the
column
contributions
were
large
in
the
original
analysis,
they
tended
to
stay
large.
We're
getting
a
relatively
consistent
story
from
the
bootstrap
forest
in
terms
of
what
columns
are
important.
Now
we
can
do
the
same
thing
with
the
boosted
tree,
and
the
results
aren't
quite
as
consistent
as
they
were
with
the
bootstrap
forest.
So
here
is
a
bunch
of
columns
where
the
initial
column
contributions
came
out very small
but
they
had
a
more
substantial
contribution
in
some
of
the
random
reshuffles
of
the
validation
column.
That
also
happened
quite
a
bit
over with
these
Columns
52
through
55
over
here.
Then
there
were
also
some
situations
where
the
original
column
contribution
was
quite
large,
and
most,
if
not
all,
of
the
other
column
contributions
found
in
the
simulations
were
smaller.
That
happens
here
with
Column
48,
and
to
some
extent
also
with
Column
11
over
here.
The
overall
conclusion
being
that
I
think
this
validation
shuffling
is
indicating
that
we
can
trust
the
column
contributions
from
the
bootstrap
forest
to
be
more
stable
than
those
of
the
boosted
tree.
Based on
this
comparison,
I
think
I
trust
the
column
contributions
from
the
bootstrap
forest
more,
and
I'm
going
to
use
the
columns
that
it
recommended
as
the
basis
for
some
other
models.
What
I'd
like
to
do
is
find
a
model
that
is
both
simpler
than
the
bootstrap
forest
model
and
performs
better
in
terms
of
validation
set
performance
for
predicting
pass
or
fail.
Before
proceeding
with
the
next
modeling
stuff,
I'm
going
to
do
something
that
I
should
have
probably
done
at
the
very
beginning,
which
is
to
take
a
look
at
the
sensors
in
a
scatterplot
matrix
to
see
how
correlated
the
sensor
readings
are,
and
also
look
at
histograms
of
them
as
well
to
see
if
they're
outlier-
prone
or heavily
skewed
or
otherwise
highly
non-
gaussian.
What
we
see
here
is
there
is
pretty
strong
multicollinearity
amongst
the
input
variables
generally.
We're
only
looking
at
a
subset
of
them
here,
but
this
high
multicollinearity
persists
across
all
of
the
sensor
readings.
This
suggests
that
for
our
model,
we
should
try
things
like
the
logistic
lasso,
the
logistic
elastic
net,
or
a logistic
ridge
regression
as
candidates
for
our
model
to
predict
pass
or
fail.
Before
we
do
that,
we
should
go
ahead
and
try
to
transform
our
sensor
readings
here
so
that
they're
a
little
bit
better-
behaved
and
more
gaussian-
looking.
This
is
actually
really
easy
in
JMP
if
you
have
all
of
the
columns
up
in
the
distribution
platform,
because
all
you
have
to
do
is
hold
down Alt
,
choose
Fit
Johnson,
and
this
will
fit
Johnson
distributions
to
all
the
input
variables.
This
is
a
family
of
distributions
that
is
based
on
a
four
parameter
transformation
to
normality.
As
a
result,
we
have
a
nice
feature
in
there
that
we
can
also
broadcast
using Alt
Click,
where
we
can
save
a
transformation
from
the
original
scale
to
a
scale
that
makes
the
columns
more
normally
distributed.
If
we
go
back
to
the
data
table,
we'll
see
that
for
each
sensor
column,
a
transform
column
has
been
added.
If
we
bring
these
transformed
columns
up
with
a
scatterplot
matrix
and
some histograms,
we
clearly
see
that
the
data
are
less
skewed
and
more
normally
distributed
than
the
original
sensor
columns
were.
Now
the
bootstrap
forest
model
that
Ron
found
only
really
recommended
a
small
number
of
columns
for
use
in
the
model.
Because
of
the
high
collinearity
amongst
the
columns,
the
subset
that
we
got
could
easily
be
part
of
a
larger
group
of
columns
that
are
correlated
with
one
another.
It
could
be
beneficial
to
find
that
larger
group
of
columns
and
work
with
that
at
the
next
modeling
stage.
An
exploratory
way
to
do
this
is
to
go
through
the
cluster
variables
platform
in JMP
.
We're
going
to
work
with
the
normalized
version
of
the
sensors
because
this
platform
is
PCA
and
factor
analysis
based,
and
will
provide
more
reliable
results
if
we're
working
with
data
that
are
approximately
normally
distributed.
Once
we're
in
the
variable
clustering
platform,
we
see
that
there
is
very
clear,
strong
associations
amongst
the
input
columns.
It
has
identified
that
there
are
seven
clusters,
and
the
largest
cluster,
the
one
that
explains
the
most
variation,
has
25
members.
The
set
of
cluster
members
is
listed
here
on
the
right.
Let's
compare
this
with
the
bootstrap
forest.
Here
on
the
left,
we
have
the
column
contributions
from
the
bootstrap
forest
model
that
you
should
be
familiar
with
by
now.
On
the
right,
we
have
the
list
of
members
of
that
largest
cluster
of
variables.
If
we
look
closely,
we'll
see
that
the
top seven
contributing
terms
all
happen
to
belong
to
this
cluster.
I'm
going
to
hypothesize
that
this
set
of
25
columns
are
all
related
to
some
underlying
mechanism
that
causes
the
units
to
pass
or
fail.
What
I
want
to
do
next
is
I
want
to
fit
models
using
the
generalized
regression
platform
with
the
variables
in
Cluster
1
here.
It
would
be
tedious
if
I
had
to
go
through
and
individually
pick
these
columns
out
and
put
them
into
the
launch
dialog.
Fortunately,
there's
a
much
easier
way
where
you
can
just
select
the
rows
in
that
table
and
the
columns
will
be
selected
in
the
original
data
table
so
that
when
we
go
into
the
fit
model
launch
dialog,
we
can
just
click
Add
and
those
columns
will
be
automatically
added
for
us
as
model
effects.
Once
I
got
into
the
Generalized
Regression
platform,
I
went
ahead
and
fit
a
lasso
model
and
elastic
net
model
and
a
ridge
model
to have
them
compared
here
to
each
other,
and
also
to
the
logistic
regression
model
that
came
up
by
default.
We're
seeing
that
the
lasso
model
is
doing
a
little
bit
better
than
the
rest
in
terms
of
its
validation
generalized
RS
quare.
The
difference
between
these
methods
is
that
there's
different
amounts
of
variable
selection
and
multicollinearity
handling
in
each
of
them.
Logistic regression
has
no
multicollinearity
handling
and
no
variable
selection.
The
lasso
is
more
of
a
variable
selection
algorithm,
although
it
has
a
little
bit
of
multicollinearity
handling
in
it
because
it's
a
penalized
method.
Ridge
regression
has
no
variable
selection
and
is
heavily
oriented
around
multicollinearity
handling.
The
elastic
net
is
a
hybrid
between
the
lasso
and
ridge
regression.
In
this
case,
what
we
really
care
about
is
just
the
model
that's
going
to
perform
the
best.
We
allow
the
validation
to
guide
us.
We're
going
to
be
working
with
the
lasso
model
from
here
on.
Here's
the
prediction
profiler
for
the
lasso
model
that
was
selected.
We
see
that
the
lasso
algorithm
has
selected
eight
sensors
as
being
predictive
of
pass
or
fail.
It
has
some
great
built-in tools
for
understanding
what
the
important
variables
are,
both
in
the
model
overall
and,
new
to
JMP
Pro
17,
we
have
the
ability
to
understand
what
variables
are
most
important
for
an
individual
prediction.
We
can
use
the
variable
importance
tools
to
answer
the
question,
"What
are
the
important
variables
in
the
model?"
We
have
a
variety
of
different
options
for
this.
We
have
a
variety
of
different
options
for
how
this
could
be
done.
But
because
of
the
multicollinearity
and
because
this
is
not
a very
large
model,
I'm
going
to
go
ahead
and
use
the
dependent
resampled
inputs
technique,
since
we
have
multicollinearity
in
the
data,
and
this
has
given
us
a
ranking
of
the
most
important
terms.
We
see
that
Column
18
is
the
most
important,
followed
by
Column
27
and
then
52,
all
the
way
down.
We
can
compare
this
to
the
bootstrap
forest
model,
and
we
see
that
there's
agreement
that
Variable
18
is
important,
along
with
52,
61,
and
53.
But
one
of
the
terms
that
we
have
pulled
in
because
of
the
variable
clustering
step
that
we
had
done,
Sensor
27
turns
out
to
be
the
second
most
important
predictive
in
this
lasso
model.
We've
hopefully
gained
something
by
casting
a
wider
net
through
that
step.
We've
found
a
term
that
didn't
turn
up
in
either
of
the
bootstrap
forest
or
the
boosted
tree
methods.
We
also
see
that
the
lasso
model
has
an
RS
quare
of
0.9,
whereas
the
bootstrap
forest
model
had
an
RS
quare
of
0.8.
We have
a
simpler
model
that
has
an
easier
form
to
understand
and
is
easier
to
work
with,
and
also
has
a
higher
predictive
capacity
than
the
bootstrap
forest
model.
Now,
the
variable
importance
metrics
in
the
profiler
have
been
there
for
quite
some
time.
The
question
that
they
answer
is,
"Which
predictors
have
the biggest
impact
on
the
shape
of
the
response
surface
over
the
data
or
over
a
region?"
In
JMP
17
Pro,
we
have
a
new
technique
called
SHAP
Values
that
is
an
additive
decomposition
of
an
individual
prediction.
It
tells
you
by
how
much
each
individual
variable
contributes
to
a
single
prediction,
rather
than
talking
about
variability
explained
over
the
whole
space.
The
resolution
of
the
question
that's
answered
by
Shapley
values
is
far
more
local
than
either
the
variable
importance
tools
or
the
column
contributions i
n
the
bootstrap
forest.
We
can
obtain
the
Shapley
Values
by
going
to
the
red
triangle
menu
for
the
profiler,
and
we'll
find
the
option
for
them
over
here,
fourth from
the
bottom.
When
we
choose
the
option,
the
profiler
saves
back
SHAP
columns
for
all
of
the
input
variables
to
the
model.
This
is,
of
course,
happening
for
every
row
in
the
table.
What
you
can
see
is
that
the
SHAP V
alues
are
giving
you
the
effect
of
each
of
the
columns
on
the
predictive
model.
This
is
useful
in
a
whole
lot
of
different
ways,
and
for
that
reason,
it's
gotten
a
lot
of
attention
in intelligible
AI,
because
it
allows
us
to
see
what the
contributions
are
of
each
column
to
a
black
box
model.
Here,
I've
plotted
the
SHAP V
alues
for
the
columns
that
are
predictive
in
the
last
fit model
that
we
just
built.
If
we
toggle
back
and
forth
between
the
good
units
and
the
units
that
failed,
we
see
the
same
story
that
we've
been
seeing
with
the
variable
importance
metrics
for
this,
that
Column
18
and
Column
27
are
important
in
predicting
pass
or
fail.
We're
seeing
this
at
a
higher
level
of
resolution
than
we
do
with
the
variable
importance
metrics,
because
each
of
these
points
corresponds
to
an
individual
row
in
the
original
dataset.
But
in
this
case,
I
don't
see
the
SHAP
Values
really
giving
us
any
new
information.
I
had
hoped
that
by
toggling
through
the
other
failure
modes,
maybe
I
could
find
a
pattern
to
help
tease
out
different
sensors
that
are
more
important
for
particular
failure
modes.
But
the
only
thing
I
was
able
to
find
was
that
Column
18
had
a
somewhat
stronger
impact
on
the
Velocity
Type
1
failure
mode
than
the
other
failure
modes.
At
this
point,
we've
had
some
success
using
those
Cluster
1
columns
in
a
binary
pass/
fail
model.
But
when
I
broke
out
the
SHAP
Values
for
that
model,
by
the
different
failure modes
I
wasn't
able
to
discern
a
pattern
or
much
of
a
pattern.
What
I
did
next
was
I
went
ahead
and
fit
the
failure
mode
response
column
test
results
using
the
Cluster
1
columns,
but
I
went
ahead
and
excluded
all
the
pass
rows
so
that
the
modeling
procedure
would
focus
exclusively
on
discerning
which
failure
mode
it is
given
that
we
have
a
failure.
I
tried
the
multinomial
lasso,
elastic
net,
and
ridge,
and
I
was
particularly
happy
with
the
lasso
model
because
it
gave
me
a
validation
RS
quare
of
about
0.94.
Having
been
pretty
happy
with
that,
I
went
ahead
and
saved
the
probability
formulas
for
each
of
the
failure
modes.
Now
the
task
is
to
come
up
with
a
simple
rule
that
post
processes
that
prediction
formula
to
make
a
decision
about
which
failure
mode.
I
call
this
the
partition
trick.
The partition trick is
where
I
put
in
the
probability
formulas
for
a
categorical
response,
or
even
a
multinomial
response.
I
put
those
probability
formulas
in
as Xs.
I
use
my
categorical
response
as
my
Y.
This
is
the
same
response
that
was
used
for
all
of
these
except
for pass,
actually.
I
retain
the
same
validation
column
that
I've
been
working
with
the
whole
time.
Now
that
I'm
in
the
partition
platform,
I'm
going
to
hit
Split
a
couple
of
times,
and
I'm
going
to
hope
that
I
end
up
with
an
easily
understandable
decision
rule
that's
easy
to
communicate.
That
may
or
may
not
happen.
Sometimes
it
works,
sometimes
it
doesn't.
So
I
split
once,
and
we
end
up
seeing
that
whenever
the
probability
of
pass
is
higher
than
0.935,
we
almost
certainly
have
a
pass.
Not
many
passes
are
left
over
on
the
other
side.
I
take
another
split.
We
find
a
decision
rule
on
ITM
that
is
highly
predictive
of
ITM
as
a
failure
mode.
Split
again.
We
find
that
whenever
Motor
is
less
than
0.945,
we're
either
predicting
Motor
or
Brake.
We
take
another
split.
We
find
that
whenever
Velocity
Type
1,
its probability
is
bigger
than
0.08
or likely
in
a
Velocity
Type
1
situation
or
in
a
Velocity T
ype
2
situation.
Whenever
Velocity
Type
1
is
less
than
0.79,
we're
likely
in
a
gripper
failure
mode
or
an
IMP
failure
mode.
What
do
we
have
here?
We
have
a
simple
decision
rule.
We're
going
to
not
be
able
to
break
these
failure
modes
down
much
further
because of
the
very
small
number
of
actual
events
that
we
have.
But
we
can
turn
this
into
a
simple
rule
for
identifying
units
that
are
probably
good,
and
if
they're
not,
we
have
an
idea
of
where
to
look
to
fix
the
problem.
We
can
save
this
decision
rule
out
as
a
leaf
label
formula.
We
see
that
on
the
validation
set,
when
we
predict
it's
good,
it's good most
of
the
time.
We
did
have
one
misclassification
of
a
Velocity
Type 2
failure
that
was
actually
predicted
to
be
good.
Predict
grippers
or
IMP,
it's
all
over
the
place.
That
leaf
was
not
super
useful.
Predicting
ITM is
100
%.
Whenever
we
predict
a
motor
or
brake,
on
the
validation
set,
we
have
a
motor
or
a
brake
failure.
When
we
predict
a
Velocity
Type
1 or 2,
it
did
a
pretty
good
job
of
picking
that
up
with
that
one
exception
of
the
single
Velocity
Type
2
unit
that
was
in
the
validation
set,
and
that one
happened
to
have
been
misclassified.
We
have
an
easily
operational
rule
here
that
could
be
used
to
sort
products
and
give
us
a
head
start
on
where
we
need
to
look
to
fix
things.
I
think
this
was
a
pretty
challenging
problem,
because
we
didn't
have
a
whole
lot
of
data.
But
we
didn't
have
a
lot
of
rows,
but
we
had
a
lot
of
different
categories
to
predict
and a
whole
lot
of
possible
predictors
to
use.
We've
gotten
there
by
taking
a
series
of
steps,
asking
questions,
sometimes
taking
a
step
back
and
asking
a
bigger
question.
Other
times,
narrowing
in
on
particular
sub-
issues.
Sometimes
our
excursions
were
fruitful,
and
sometimes
they
weren't.
Our
purpose
here
is
to
illustrate
how
you
can
step
through
a
modeling
process,
through
this
sequence
of
asking
questions
using
modeling
and
visualization
tools
to
guide
your
next
step,
and
moving
on
until
you're
able
to
find
a
useful,
actionable,
predictive
model.
Thank
you
very
much
for
your
attention.
We
look
forward
to
talking
to
you
in
our
Q&A
session
coming
up
next.
