Hi,
my
name
is
Clay
Barker,
and
I'm
a
statistical
developer
in
the
JMP
group.
I'm
going
to
be
talking
about
some
new
features
in
the
distribution
platform
geared
towards
analyzing
limit
of
detection
data.
It's
something
I
worked
on
with
my
colleague,
Laura
Lancaster,
who's
also
a
stat
developer
at
JMP.
To
kick
things
off,
what
is
a
limited
detection
problem?
What
are
we
trying
to
solve?
It
is
basic
level,
a
limit
of
detection
problem
is
when
we
have
some
measurement
device
and
it's
not
able
to
provide
good
measurements
outside
of
some
threshold.
That's
what
we
call
a
limit
of
detection
problem.
F
or
example,
let's
say
we
have,
we're
taking
weight
measurements
and
we're
using
a
scale
that's
not
able
to
make
measurements
below
1g
.
In
this
instance,
we'd
say
that
we
have
a lower
detection
limit
of
1
because
we
can't
measure
below
that.
But
in
our
data
set,
we're
still
recording
those
values
as
1.
Because
of
those
limitations,
we
see,
we
might
have
a
data
set
that
looks
like
this.
In
part,
we
have
some
values
of
1
and
some
non-1
values
that
are
much
bigger.
We
don't
really
believe
that
those
values
are
1s.
We
just
know
that
those
are
at
most
1.
This
data
happens
all
the
time
in
practice.
If
you
think
about
sometimes
we're
not
able
to
measure
below
a
lower
threshold.
Sometimes
we're
not
able
to
measure
above
an
upper
threshold.
Those
are
both
limited
detection
problems.
What
should
we
do
about
that?
Let's
look
at
a
really
simple
example.
I
simulated
some
data
that
are
normally
distributed
with
mean
10
and
variance 1
,
and
we're
imposing
a
lower
detection
limit
of
9.
If
we
look
at
our
data
set
here,
we
have
some
values
above
9,
and
we
have
some
9s.
When
we
look
at
the
histogram,
this
definitely
doesn't
look
normally
distributed
because
we
have
a
whole
bunch
of
extra
9s
and
we
don't
have
anything
below
9.
What
happens
if
we
just
model
that
data
as
is?
Well,
it
turns
out
the
results
aren't
that
great.
We
get
a
decent
estimate
of
our
location
parameter,
our
Mu,
it's
really
close
to
10,
which
is
the
truth.
But
we've
really
underestimated
that
scale
or
dispersion
parameter.
We've
estimated
it
to
be
0.8,
when
the
truth
is
that
we
generated
it
with
scale
equal
to
1.
You'll
notice
that
our
confidence
interval
for
that
scale
parameter
doesn't
even
cover
1.
It
doesn't
contain
the
truth
and
that's
generally
not
a
great
situation
to
be
in.
What's
more,
if
we
look
at
theâ¦
We
fit
a
handful
of
distributions,
we
fit
the
log
normal,
the
gamma,
and
the
normal,
well the
normal
distribution,
which
is
what
we
use
to
generate
our
data,
it
isn't
even
competitive
based
on
the
AIC.
Based
on
those
AIC
values,
we
would
definitely
choose
a
log
normal
distribution
to
model
our
response.
We
haven't
done
a
good
job
estimating
our
parameters.
We're
not
even
choosing
to
use
the
distribution
that
we
generated
the
data
with.
W
e
just
threw
all
those
9s
into
our
data
set.
We
ignored
the
fact
that
that was
incomplete
information
and that
didn't
work
out
well.
What
if,
instead
of
ignoring
that
limit
of
detection,
what
if
we
just
throw
out
all
those
times?
Well,
now
we've
got
a
smaller
data
set
and
it's
biased.
We've
thrown
out
a
large
chunk
of
our
data
on
it.
We
have
a
biased
sample
now.
Now
if
we
fit
our
normal
distribution,
now
we're
overestimating
the
location
parameter,
and
we're
still
underest
imating
the
scale
parameter.
We're
actually
in
quite
a
bad
position
still,
because
we
haven't
done
a
good
job
with
either
of
those
parameters.
We're
still
unlikely
to
pick
the normal
distribution.
Based
on
the
AIC,
the
log
normal
and
the
gamma
distribution
both
fit
quite
a
bit
better
than
the
normal
distribution.
We're
still
in
a
bad
place.
We
tried
throwing
out
the
9
s,
and
that
didn't
turn
out
well.
We
tried
just
including
them
as
9
s.
That
didn't
turn
out
well
either.
What
should
we
do
instead?
The
answer
is
that
we
should
treat
those
observations
at
the
limit
of
detection.
We
should
treat
those
as
censored
observations.
Censoring
is
a
situation
where
we
only
have
partial
information
about
our
response
variable.
That's
exactly
the
situation
we're
in
here.
If
we
have
an
observation
at
the
lower
detection
limit,
and
here
I've
denoted
it
D sub L,
we
say
that
observation
is
less
censored.
We
don't
say
that
Y
is
equal
to
that
limit
of
detection.
We
say
that
Y
is
less
than
or
equal
to
that
DL
value.
On
the
flip
side,
if
we
have
a
upper
limit
of
detection,
denoted
DU
here,
those
observations
are
right
censored.
Because
we're
not
saying
that
Y
is
equal
to
that
value.
We're
just
saying
it's
at
least
that
value.
If
you're
looking
for
more
information
about
how
to
handle
censored
data,
one
of
the
references
that
we
suggest
all
the
time
is
Meeker and
Escobar's
book
Statistical
Methods
for
Reliability
Data.
That's
a
really
good
overview
for
how
you
should
treat
censored
data.
If
you've
used
some
of
the...
If
you
use
some
of
the
features
and
the
survival
and
reliability
menu
in JMP,
then
you're
familiar
with
things
like
life
distribution
and
fit
life
by
X.
These
are
all
platforms
that
accommodate
censoring
in
JMP.
What
we're
excited
about
in
JMP
17
is
now
we
have
added
some
features
to
distribution
so
that
we
can
handle
this
limit
of
detection
problem
and distribution
as
well.
All
you
have
to
do
is
you
add
a
detection
limit
column
property
to
your
response
variable,
and
you
specify
what
the
upper
and
or
lower
detection
limit
is,
and
you're
good
to
go,
there's
nothing
else
you
have
to
do.
In
my
simulated
example,
I
had
a
lower
detection
limit
of
9.
I
would
put
9
in
the
lower
detection
limit
field
here.
That's
really
all
you
have
to
do.
By
specifying
that
detection
limit,
now
distribution
is
going
to
say,
okay,
I
know
that
values
of
9
are
actually
left
censored,
and
I'm
going
to
do
estimation
accounting
for
that.
Now
with
that
same
simulated
example,
and
this
lower
detection
limit
specified,
now
you'll
notice
we
get
a
much
more
reasonable
fit
the
normal
distribution.
Now
our
confidence
interval
for
both
the
location
and
scale
parameter
covers
the
truth,
because
we
know,
again,
the
location
was
10
and
the
scale
was
1.
Now
our
confidence
intervals
cover
the
truth
and that's
a
much
better
situation.
If
you
look
at
the
CDF
plot
here,
this
is
a
really
good
way
to
compare
our
fitted
distribution
to
our
data.
What
it's
doing
is
that
red
line
is
the
empirical
CDF,
and
the
green
line
is
the
fitted
normal
CDF.
as
you
can
tell,
they're
really
close
up
until
9.
And that
makes
sense,
because
that's
where
we
have
censoring.
We're
doing
a
much
better
job
fitting
these
data
because
we're
properly
handling
that
detection
limit.
I
just
wanted
to
point
out
that
when
you've
specified
the
detection
limit,
the
report
makes
it
really
clear
that
we've
used
it.
As
you
can
see
here,
it
says
the
fitted
normal
distribution
with
detection
limits,
and
it
lets
you
know
exactly
which
detection
limits
it
used.
Now
not
only
are,
because
we're
doing
a
better
job
estimating
our
parameters,
things
like
inference
about
those
parameters
is
more
trustworthy.
If
we
do
something
like
we
look
at
the
distribution
profiler
now
we
can
trust
these
inference
based
on
our
fitted
distribution,
we
feel
much
better
about
trusting
things
like
the
distribution
profiler.
With
the
simulated
example,
if
we
use
our
fitted
normal
distribution,
Because
we
properly
handled
censoring,
we
know
that
about
16
%
of
the
observations
are
falling
below
that
lower
detection
limit.
I
also
wanted
to
point
out
that
when
you
have
detection
limits
in
distribution,
now
we're
only
able
to
fit
a
subset
of
the
distributions
that
you
would
normally
see
in
the
distribution
platform.
We
can
fit
the
normal,
exponential,
gamma
log,
normal,
WI
and
beta.
All
of
those
distributions
support
censoring
or
limited
detection
in
distribution.
But
if
you
were
using
something
like
the
mixture
of
normals,
well,
that
that
doesn't
extend
well
to
sensor
data.
You're
not
going
to
be
able
to
fit
that
when
you
have
a
limit
of
detection.
I
also
wanted
to
point
out
if
you
have
JMP
pro
and
you're
used
to
using
the
generalized
regression
platform,
generalized
regression
recognizes
that
exact
same
column
property.
The
detection
limit
column
property
is
recognized
by
both
distribution
and
generalized
.
One
of
the
really
nice
things
about
this
new
feature
is
that
it
gets
carried
on
to
the
capability
platform.
If
you
do
your
fit
and
distribution,
and
you
launch
capability,
now
we're
going
to
get
more
trustworthy
capability
results.
Let's
say
that
we're
manufacturing
a
new
drug,
and
we
want
to
measure
the
amount
of
sum
impurity
in
the
drug.
Our
data
might
look
like
what
we
have
here.
We
have
a
bunch
of
small
values,
and
we
have
a
lower
detection
limit
of
1
mg.
these
values
of
1
that
are
highlighted
here,
we
don't
really
think
those
are
1.
We
actually
think
it's
something...
We
know
that
it's
something
less
than 1
.
We
have
an
upper
specification
limit
of
2.5
milligrams.
this
is
a
situation
where
we
have
both
spec
limits
and
detection
limits.
It's
really
easy
to
specify
those
in
the
column
properties.
Here
we've
defined
our
upper
spec
limit
as
2.5
And
our
lower
detection
limit
of
1.
Now
all
you
have
to
do
is
just
give
distribution
the
column
that
you
want
to
analyze.
It
knows
exactly
how
to
handle
that
response
variable.
Let's
look
at
the
capability
results.
Now,
because
we've
properly
handled
that
limit
of
detection,
we
trust
that
our
log
normal
fit
is
good.
We
see
that
our
Ppk,
value
here
is
0.546
.
That's
not
very
good.
Usually
you
would
want
a
Ppk
above
1.
This
is
telling
us
that
our
system
is
not
very
capable.
We've
got
some
problems
that
we
might
need
to
sort
out.
Once
again,
what
would
have
happened
if
we
had
ignored
that
limit
of
detection
and
we
had
just
treated
all
those
1s
as
if
they
truly
were
1s.
Well,
let's
take
a
look.
We
do
our
fit,
ignoring
the
limit
of
detection,
and
we
get
a
Ppk
of
above
1.
Based on
this
fit,
we
would
say
that
we
actually
have
a
decently
capable
system,
because
a
Ppk
of
1
is
not
too
bad.
It
might
be
an
acceptable
value.
By
ignoring
that
limit
of
detection,
we've
tricked
ourselves
into
thinking
our
system
is
more
capable
than
it
really
is.
I
think
this
is
a
cool
example,
because
we
have
a
lower
detection
limit,
which
may
lead
you
to
believe,
well,
I
might
be
maybe
ignoring
the
limit
of
detection
would
be
conservative,
because
I'm
overestimating
the
location
parameter.
That's
true,
when
we
ignore
the
limit
of
detection,
we're
overestimating
that
location
parameter.
But
the
problem
is
we're
grossly
underestimating
the
scale
parameter.
That's
what
makes
us
make
bad
decisions
out
in
the
tail
of
that
distribution.
By
ignoring
that
limit
of
detection,
we've
really
gotten
ourselves
into
a
bad
situation.
Just
to
summarize,
it's
really
important
to
recognize
when
our
data
feature
a
limit
of
detection.
I
think
it's
easy
to
think
of,
sometimes
we
think
about
data
sets
where
maybe
we've
analyzed
the
response
as
is
in
the
past,
when
really,
maybe
we
should
have
adjusted
for
a
limit
of
detection.
Because
like
we
just
saw,
when
we
ignore
those
limits,
we
get
misleading
fits.
Then
we
may
make
bad
decisions
based
on
those
misleading
fits.
Like
we
saw
in
our
example,
we
got
bad
estimates
or
the
location
and
scale
parameters,
and
our
Ppk
estimate
was
almost
double
what
it
should
have
been.
But
what
we're
excited
about
in
JMP
17
is
that
the
distribution
platform
makes
it
really
easy
to
avoid
these
pitfalls
and
to
analyze
this
kind of data
properly.
All
you
have
to
do
is
specify
that
detection
limit
column
property,
and
distribution
knows
exactly
what
to
do
with
that.
Today
we
only
looked
at
lower
detection
limits,
but
you
can
just
as
well
have
upper
detection
limits
as
well.
In
fact,
you
can
have
both.
Like
I
said,
there's
only
six
distributions
that
currently
support
censoring
inside
of
the
distribution
platform.
But
those
are
also
the
six
most
important
distributions
for
these
kinds
of
data.
It
really
is a
good
selection
of
distributions.
That's
it.
I just wanted
to
thank
you
for
your
time
and
encourage
you
to
check
out
these
enhancements
to
the
distribution
platform.
Thanks.
