Hi,
everyone,
my
name
is
Di
Michelson.
I'm
an
Instructor
in
the
JMP
Education
group.
My
co-authors
for
this
presentation
are
Jordan
Hiller
and
Byron
Wingerd,
who
are
both
JMP
Systems
Engineers.
We're
here
today
to
warn
you
of
the
dangers
in
using
JMP's
Levey-
Jennings
control
charts.
We
wanted
to
present
at
Discovery
because
we
believe
Levey-Jennings
charts
are
being
misused,
resulting
in
missed
signals.
Today,
I'll
start
with
a
quick
control
chart
review
followed
by
a
quick
history,
and
that
will
lead
into
how
we've
seen
Levey-Jennings
charts
being
misused.
We'll
conclude
with
some
recommendations
for
use
of
these
charts
and
some
final
thoughts.
Now,
every
quality
system
has
a
bit
of
process
control
in
it.
These
are
the
DMAIC
steps
from
Six
Sigma,
but
you
might
be
using
QbD
or
another
quality
system.
Control
charts
are
used
at
the
end
of
an
improvement
process
when
you've
put
the
process
on
target
and
minimize
the
variability.
The
process
is
now
stable,
which
means
the
probability
distribution
is
not
changing
over
time.
Now
it's
time
to
monitor
that
process
variability
to
verify
that
the
distribution
stays
the
same
and
is
not
changing
in
the
future.
What
is
a
control
chart?
A
control
chart
is
simply
a
run
chart
of
the
data
with
the
addition
of
control
limits.
The
center
line
is
based
on
the
historical
mean
of
the
process,
and
the
width
of
the
control
limits
is
based
on
the
historical
variation
of
the
process.
If
the
process
distribution
does
change,
observations
will
eventually
fall
outside
of
the
control
limits,
signaling
to
the
process
owner
to
take
action
to
assign
a
cause
to
the
out-of-control
point
and
restore
the
process
to
a
state
of
control.
Many
types
of
control
charts
have
been
developed
for
different
situations.
XM-R
charts
or
individual
and
moving
range
charts
for
data
collected
one
point
at
a
time.
Xbar-R
or
Xbar-S
charts
for
summarized
data
collected
multiple
points
at
a
time.
CUSUM
or
EWMA
control
charts
to
detect
small
shifts,
and
even
m
odel-driven
multivariate control charts
to
control
multiple
correlated
variables
on
one
chart.
To
find
those
all
important
control
limits,
you
have
to
answer
this
question,
how
do
you
estimate
the
historical
process
distribution?
First,
you
have
to
determine
if
the
process
is
stable
enough
for
control
charting.
Or
does
it
need
active
process
improvement?
This
is
called
Phase
I.
It's
an
active
phase
of
data
collection
and
sampling
plan
adjustment,
limit
calculation,
maybe
even
some
statistical
modeling.
This
phase
ends
when
you
have
collected
enough
data
to
determine
that
the
process
is
stable.
Then
the
SPC
system
shifts
into
Phase
II.
The
sampling
plan
and
the
control
limits
are
fixed
and
new
data
are
judged
by
the
fixed
limits.
There's
many
considerations
for
Phase
I
data
collection.
The
sampling
plan,
including
rational
sampling
and
rational
subgrouping,
the
sample
size,
how
much
data
do
you
need
to
collect
to
fix
those
limits,
the
type
of
data
that
you're
collecting,
the
type
of
control
chart
that
you
want
to
make.
Rather
than
talking
about
all
these
in
detail
here,
I
will
refer
you
to
our
excellent,
free,
online,
self-paced,
e-learning
course
called
Statistical
Process
Control.
It's
available
in
the
JMP
Community.
You'll
often
hear
that
control
chart
limits
are
three-
sigma
limits.
This
means
that
we
need
to
estimate
the
standard
deviation
of
the
process
from
our
data,
multiply
by
three,
and
then
add
and
subtract
from
the
estimate
of
the
mean
of
the
process.
When
Walter Shewhart
developed
the
first
control
charts
in
the
1920s,
he
realized
that,
like
other
statistical
procedures,
he
needed
to
test
for
signals
using
the
noise.
He
needed
to
compare
the
between
subgroup
variation
using
the
within
subgroup
variation.
This
led
him
directly
to
X
bar-
R
charts,
where
the
average
of
the
within
subgroup
range
is
used
as
an
estimate
of
sigma.
Specifically,
it
led
him
away
from
the
naive
thinking
that
we
should
use
X
bar
plus
or
minus
three
times
the
sample
standard
deviation.
Don't
estimate
sigma
using
s for
control
charts.
Let
me
say
that
again.
Don't
estimate
sigma
using
the
sample
standard
deviation.
Oh,
and
here
it
is
in
red.
Don't
use
the
sample
standard
deviation
to
estimate
sigma.
Why
not?
The
purpose
of
control
charts
is
to
detect
a
signal
from
your
process,
detect
a
change
in
the
process
distribution.
Using
the
sample
standard
deviation
aggregates
all
the
data,
including
a
potential
signal,
and
this
will
inflate
the
estimate
of
sigma
and
you
will
miss
signals.
That
takes
us
back
to
the
beginning.
If
the
purpose
is
to
detect
signals,
why
would
you
make
a
chart
that
makes
you
miss
signals?
Just
don't
do
it.
But
the
default
method
for
estimating
sigma
in
JMP's
Levey-Jennings
control
chart
is
to
use
the
sample
standard
deviation.
The
Levey-Jennings
chart
was
developed
for
a
very
specific
situation
in
clinical
chemistry,
one
where
an
analytical
method
is
validated
using
a
gage
study
with
factors
that
are
relevant
to
that
process.
The
data
are
collected
according
to
a
specified
experimental
design.
Here
are
some
possible
random
and
fixed
factors
in
the
experiment.
The
variance
components
for
the
random
factors
are
calculated
using
a
statistical
model.
Then
the
total
variability
is
calculated
from
the
variance
components
and
used
as
the
estimate
of
sigma
for
the
Levey-Jennings
control
chart.
This
chart
has
a
specific
purpose
in
clinical
chemistry.
It
can
be
used
whenever
you
have
an
external
estimate
of
sigma.
How
did
we
get
here
where
these
charts
are
being
used
for
situations
in
industries
other
than
the
one
I
just
described?
The
history
of
SPC
started
about
100
years
ago
when
Walter Shewhart
developed
the
first
control
chart.
He
figured
out
to
use
the
within
or
the
short-term
variation
using
subgroups
with
multiple
items.
But
he
couldn't
figure
out
how
to
estimate
short-term
variation
for
subgroups
of
size
1.
There
is
no
within.
In
1950,
Levey
and
Jennings
published
their
paper,
and
they
did
not
use
s to
estimate
sigma.
Their
paper
introduced
control
charts
to
clinical
chemistry
and
used Shewhart's
Xbar-
R chart
with
two
replicates,
so
subgroup
size
2.
Now,
just
after
that,
in
the
early
1950s,
Tippett
discovered
that
you
can
use
the
average
moving
range
of
individual
values
as
a
short-term
estimate
of
sigma.
It's
not
perfectly
a
within
estimator,
but
it's
short-term
enough
that
you're
not
aggregating
over
all
the
data.
Very
quickly,
ASTM
and
Western
Electric
published
books
and
standards
using
Tippett's
X-
MR
chart.
But
clinical
chemistry
didn't
hear
about
these
charts.
In
1952,
Henry
and
Segalove
published
a
paper
using
s
to
estimate
sigma
using
subgroups
of
size
1.
Don
Wheeler
indicates
that
XM-R
charts
were
not
used
very
often
until
about
1980.
He
introduced
them
to
Deming,
and
Wheeler
and
Deming
taught
thousands
of
clients
and
popularized
the
XM-R
chart.
It's
really
interesting
to
me
that
they
weren't
popular
back
then
because
in
my
experience,
which
started
in
the
early
'90s,
charts
for
subgroups
of
size
1
are
much
more
popular
than
Xbar
charts
are.
All
right,
just
as
the
XM-R
charts
were
gaining
popularity
in
manufacturing,
James
Westgard
published
his
Westgard
rules,
and
wrote
his
popular
book,
Basic
QC
Practices
for
clinical
chemistry.
He
calls
the
control
charts,
Levey-Jennings.
They
were
added
to
JMP
in
version
5
in
2003
by
customer
request.
Now,
ever
since
Jordan
and
Byron
and
I
have
been
working
at
JMP,
we've
seen
examples
from
all
industries
of
the
misuse
of
the
default
Levey-Jennings
chart.
But
now
you
know
better,
and
you
will
never
use
the
sample
standard
deviation
to
estimate
sigma
for
control
charts
ever
again,
right?
All right,
I'm
going
to
turn
it
over
to
Jordan,
and
he'll
show
you
in
JMP
the
problems
with
misusing
these
charts.
Thanks,
Di.
We
created
a
simulation
in
JMP
to
show
how
the
Levey-Jennings
chart,
when
misused,
will
definitely
lead
to
missed
signals.
You
can
download
this.
It's
a
JMP
add-in
that's
saved
in
the
JMP C
ommunity
with
the
other
presentation
materials.
Here
we
show
two
control
charts
on
the
same
data
set.
Here's
the
individual
chart
on
top,
and
the
Levey-Jennings
chart
on
the
bottom.
Just
a
brief
note,
the
individual's
chart
is
usually
displayed
with
the
moving
range
chart
underneath
it.
We're
omitting
that
for
visual
simplicity
here.
I'll
also
mention
that
the
Levey-Jennings
chart
is
drawn
in
the
way
we're
warning
against.
That
is,
these
are
not
historical
limits,
these
are
limits
calculated
from
the
data
in
the
chart
itself.
The
30
data
points
that
we
are
graphing
here
are
drawn
randomly
from
a
standard
normal
distribution.
That
is,
the
mean
is
zero,
standard
deviation
is
one.
Whenever
I
click
this
new
data
button
over
here,
we'll
get
a
new
random
sample.
Because
we
know
the
true
population
parameters,
we
can
tell
how
accurately
these
charts
perform.
The
true
population
sigma
is
one,
and
we're
going
to
hope
that
the
sigma
estimates
from
the
two
charts
are
close
to
that.
The
data
is
stable,
it's
from
a
normal
distribution.
Whenever
we
see
a
point
outside
the
control
limits,
it's
a
false
positive
here.
In
a
simulation
of
5,000
data
sets,
the
individual's
chart
has
a
false
positive
about
7%
of
the
time,
while
the
Levey-Jennings
has
one
about
3.5
%
of
the
times.
This
small
difference,
well,
it's
double,
but
this
difference
in
the
false
positive
rates,
we
think
is
one
of
the
reasons
that
folks
have
tended
to
prefer
the
Levey-Jennings
to
the
individual's
chart
sometimes.
But
here's
the
problem.
What
happens
when
we
have
some
signals
in
this
data?
We're
going
to
talk
with
a
couple
of
different
kinds
of
signals.
Let's
start
with
a
shift.
A
shift
is
an
abrupt
change
in
the
mean
of
the
process.
Let's
start
by
looking
at
a
shift
of
three
standard
deviations,
and
I'm
going
to
hit
the
new
data
a
couple
of
times.
Remember,
the
purpose
of
a
control
chart
is
to
distinguish
the
signal
from
the
noise
and
to
detect
signals.
As
the
signal
gets
larger,
we
should
be
more
likely
to
detect
it,
and
the
sigma
estimates
should
remain
close
to
the
true
value
of
one.
The
individual's
chart
does
exactly
what
we
expect.
It
usually
detects
the
shift,
and
the
sigma
estimate
for
the
individual
chart
stays
close
to
one.
The
Levey-Jennings,
not
so
much.
It
conflates
the
signal
with
the
noise.
The
sigma
estimate
is
inflated,
and
signals
generally
can't
be
detected.
How
big
does
this
shift
have
to
be
before
the
Levey-Jennings
chart
is
going
to
detect
it?
Well,
I'm
just
kidding.
It's
a
trick
question.
The
Levey-Jennings
chart
is
never
going
to
detect
a
shift
in
this
scenario.
I
can
jack
it
up
to
50
standard
deviations,
and
the
larger
the
shift
that
we
induce,
the
more
that
sigma
estimate
is
inflated
and
will
never
detect
a
shift.
Let's
see,
we
did
some
simulation
work
to
show
the
differences
between
the
performance
of
the
individual's
chart
that's
in
red
and
the
Levey-Jennings
chart
in
blue.
We're
looking
at
a
shift
that
occurs
after
the
20th
data
point
in
the
series
of
30
data
points.
That's
what
we're
looking
at
in
the
demo
I
just
showed.
In
that
situation,
a
shift
after
the
20th
data
point,
you
can
see
the
performance
of
both
of
these
charts
as
the
shift
gets
larger.
Individual's
chart
does
exactly
what
we
hope
it
would
do.
As
the
shift
becomes
larger,
it's
easier
to
detect
probability
of
having
an
alarm
is
greater.
The
Levey-Jennings
performs
terribly
as
you
saw,
and
paradoxically,
as
the
shift
gets
larger,
the
probability
of
detecting
it
approaches
zero.
Let's
take
a
look
at
that
here
in
the
demo.
We
can
change
the
shift
location,
right?
The
shift
was
after
the
20th
data
point.
But
you'll
see
that
as
the
shift
approaches
the
end
of
the
series,
and
we'll
put
it
at
29,
the
last
data
point
here.
When
the
shift
occurs
at
the
last
data
point,
both
of
the
charts
perform
similarly.
In
other
words,
if
you
are
running
your
Levey-Jennings
chart
after
every
additional
data
point,
you'll
probably
detect
most
signals.
And
this
is
to
be
expected.
If
the
shift
occurs
later,
there's
less
opportunity
for
that
signal
to
contaminate
the
noise
estimate,
and
the
sigma
estimate
is
going
to
be
accurate.
Here's
what
that
looks
like
in
the
simulations.
As
that
shift
gets
later
and
later
in
the
series,
the
performance
of
the
Levey-Jennings
chart
in
blue,
approaches
the
performance
of
the
individual's
chart.
That's
the
story
with
shifts.
It's
even
more
alarming
with
drifts.
I'm
going
to
reduce
the
shift
size
to
zero.
As
we
introduce
drift
into
the
data,
drift
is
a
gradual
linear
trend
as
the
mean
of
the
process
moves
up
in
a
consistent
way.
That's
drift.
No
matter
how
large
the
drift
that
I
induce
here,
it
will
never
be
detected
by
the
Levey-Jennings
chart.
Just
can't
be
detected.
Okay.
N
ext
let's
consider
a
batched
process.
Here,
we're
going
to
simulate
batches
with
size
six,
five
batches
here
in
the
data.
Some
folks
use
the
Levey-Jennings
chart
as
a
way
to
avoid
having
an
alarm
that's
due
to
expected
batch
variation.
Well,
it's
easy
to
see
the
problem
with
this
approach.
When
we
overlay
a
shift
or
a
drift
on
top
of
that
batch
effect,
the
Levey-Jennings
chart
is
still
never
going
to
detect
it.
I'll
add
a
shift
too.
Yeah,
the
Levey-Jennings
chart
is
insensitive,
too
insensitive,
and
the
individual's
chart
is
too
sensitive.
Neither
of
the
charts
is
really
great
for
this
situation.
Di
is
going
to
talk
about
this
in
a
few
moments.
Before
I
turn
it
back
to
Di,
let
me
show
you
what
a
Levey-Jennings
chart
done
right
looks
like.
This
is
a
demo
that
was
generated
by
my
colleague,
Byron,
using
data
that
come
from
the
Westga
rd
website,
and
let's
just
talk
about
maybe
this
top
chart
here.
We're
showing
a
chart
based
on
a
data
series.
We
have
28
data
points.
The
mean
is
198.75
in
this
sample.
The
standard
deviation
is
5.9
in
this
sample.
The
point
is,
we
are
in
Phase
II,
and
as
such,
we
have
set
the
control
limits
using
a
historical
estimate
of
sigma.
This
protects
us
against
all
of
the
problems
that
we
were
just
discussing.
It's
a
stable
estimate
of
sigma,
and
it
will
not
be
contaminated
by
any
potential
signals
in
the
data
set.
Okay,
that
is
all
I
have.
Di,
back
to
you.
All
right.
Thank
you,
Jordan,
for
that
alarming
demo.
When
Jordan
and
I
and
Byron
thought
that
it
was
time
that
someone
gave
this
talk,
we
asked
our
JMP
friends
if
they
were
seeing
what
we
were
seeing
from
talking
with
customers,
and
that
is,
misapplication
of
Levey-Jennings
charts.
What
we
found
was
that
these
charts
were
commonly
used
for
specific
situations.
The
first
one
is
what
Jordan
was
talking
about,
that
batch
processing.
It's
called
short-
run
SPC,
where
you
might
have
batches
that
change
frequently,
and
each
batch
has
a
new
mean,
maybe
even
a
new
standard
deviation.
Using
the
sample
standard
deviation
to
estimate
sigma
will
contain
batch
shifts
as
well
as
within-
batch
noise.
Like
you
saw,
you
won't
be
able
to
detect
shifts
within
a
batch
very
easily.
Instead,
you
can
use
a
chart
that
plots
the
difference
or
the
standardized
difference
from
the
batch
target
or
the
batch
mean.
The
second
situation
is
autocorrelated
processes,
and
this
is
where
observations
taken
close
together
are
more
similar
than
observations
taken
further
apart
in
time.
This
behavior
is
often
seen
when
measurements
are
taken
very
frequently
in
a
continuous
process.
Using
the
sample
standard
deviation,
again,
includes
both
noise
and
known
process
drift
due
to
the
auto
correlation
in
the
estimate
of
sigma.
There
are
a
few
ways
to
deal
with
autocorrelated
data,
including
reducing
the
sampling
frequency
if
you
can,
or
using
a
different
type
of
chart
if
you
can't.
Charts
like
CUSUM
and
EWMA
can
be
adapted
for
autocorrelation,
or
you
could
try
to
model
the
autocorrelation
and
then
use
control
charts
on
the
residuals
from
the
model.
The
residuals
also
contain
information
about
process
shifts,
and
they
should
be
uncorrelated.
F
inally,
we've
heard
a
lot
of
people
say
that
using
the
sample
standard
deviation
is
useful
because
it
gives
wide
limits
that
are
able
to
detect
huge
shifts,
but
not
too
wide
to
detect
small
and
moderate
shifts.
In
that
case,
we
recommend
that
you
just
make
up
some
limits
and
don't
advertise
that
you're
using
control
charts
because
control
charts
don't
ever
use
the
sample
standard
deviation
to
estimate
sigma.
Remember
that
West
gard's
wonderful
book
is
called
Basic
QC
Practices
,
not
Basic
SPC
Practices.
Feynman's
wonderful
quote
is
applicable
here.
We've
also
heard
some
arguments
about
Levey-Jennings
charts.
They
are
more
forgiving
than
Shewhart's
charts.
Of
course,
they
are.
I
like
this
one.
The
range
charts
were
optimized
for
hand
calculation,
and
we've
got
computers.
Why
aren't
we
calculating
standard
deviation?
As
we've
seen,
it's
not
range
versus
standard
deviation,
it's
which
standard
deviation?
You
should
always
choose
within
variability
when
you
calculate
three-sigma
limits.
Aggregate
over
noise,
not
signal
or
potential
signal.
Why
use
an
estimate
of
sigma
when
we
can
just
calculate
sigma?
Oh,
this
one
hurts
me.
This
comes
from
a
terminology
issue.
We
may
say
three-
sigma
limits,
but
we
don't
know
sigma
and
we
have
to
estimate
sigma
from
data.
This
is
the
way.
I
inherited
this
system,
and
my
boss
says
I
have
to
do
it
this
way.
Well,
can
you
find
a
new
boss?
Really,
this
is
my
Oprah
moment.
Maya
Angelou
said,
"In
the
past,
you
did
the
best
with
what
you
had,
and
now
that
you
know
better,
you
will
do
better."
I
hope
that
you
can
educate
others
to
do
better
as
well.
Thank
you
for
listening.
I've
got
some
references
for
you
here
at
the
end
of
the
presentation.
We'd
like
to
leave
you
with
two
thoughts.
First,
don't
use
the
Levey-Jennings
charts
as
they
have
been
defined
in
the
modern
world.
The
purpose
of
a
control
chart
is
to
detect
process
changes,
and
those
changes
are
found
by
comparing
signal
to
noise.
Use
of
the
sample
standard
deviation
to
estimate
sigma
inflates
that
noise
and
it
will
obscure
any
signals.
The
second
thought
is
that
when
you
are
in
Phase
I,
use
XM-R
charts,
individual
moving
range
charts
instead
of
Levey-Jennings.
When
you
move
to
Phase
II
for
ongoing
process
control,
fix
those
limits.
A
sign
of
a
stable
process
is
that
the
sigma
estimate
from
using
the
average
moving
range
is
similar
to
the
sigma
estimate
from
Levey-Jennings.