Hello,
my
name
is
Caleb
King.
I'm
a
senior
research
statistician
developer
here
in
the
Design
of
Experiments
and
Reliability
group
at
JMP
Statistical
Discovery.
Its
quite
a
mouthful.
A
s
you
probably
guess
from
the
title
of
my
talk,
I'm
going
to
be
talking
about
autonomous
vehicles,
and
specifically
how
we
can
probably
assess
the
reliability
of
autonomous
vehicles
using
some
of
the
reliability
platforms
like
JMP,
so
that
may
be
one
day
we'll
be
like
this
fine
gentleman
here
relaxing
in
our
autonomous
vehicles
as
they
take
us
wherever
we
want
to
go
without
being
scared
that
we
won't
end
up
there.
L
et's
get
into
that.
O
f
course,
autonomous
vehicles
are
fast
becoming
a
reality.
We
have
a
lot
of
companies
now
involved
in
extensive
testing
of
these
vehicles.
Now,
of
course,
some
of
that
testing
early
on
is
basic
testing
of
the
software,
testing
in
simulated
environments,
but
ultimately,
in
order
to
have
a
very
reliable
vehicle,
you
need
to
test
it
out
on
the
roads,
and
so
to
help
out
with
that,
a
couple
of
years
back,
the
California
Department
of
Motor
Vehicles
put
together
a
testing
program
that
would
allow
these
companies
to
opt
in
and
test
their
vehicles
on
their
roads.
As
part
of
that
arrangement,
though,
every
year
these
companies
have
to
provide
a
detailed
report
on
any
disengagement,
and
heaven
forbid,
crashes
involving
their
autonomous
vehicles
to
the
DMV.
Now,
because
the
DMV
is
a
federal
institution,
these
reports
are
actually
publicly
available
upon
request.
In
fact,
I'll
pull
up
a
link
here
real
quick
to
the
California
DMV
website
where
you
would
be
able
to
go
to.
In
this
case,
you
could
access
reports
from
the
last
two
years.
If
you
wanted
more,
you
just
send
an
email
to
this
link
there
pretty
quick
on
getting
back
to
you.
Now,
already
a
lot
of
people
are
using
this
data
in
academic
research.
As
a
quick
example,
I'll
click
on
this
link
here
to
a
paper
on
a
project
that
I
was
personally
involved
in
with
my
former
advisor
and
one
of
his
new PhD
students,
in
this
case,
analyzing
it
using
what's
called,
recurrent
events
data.
I'll
talk
a
little
bit
about
that
more
later.
But
again,
there's
already
research
looking
at
how
can
we
use
this
data
to
assess
the
reliability
of
autonomous
vehicles,
maybe
use
it
to
introduce
new
methods
and
so
forth.
I
t's
already
been
starting
to
be
widely
used.
Now
for
this
particular
talk,
I'll
be
looking
at
disengagement
reports,
using
that
as
a
way
to
measure
reliability
because
of
course,
we
can't
access
the
actual
software,
that's
all
proprietary
for
the
company.
But
through
the
use
of
these
disengagement
events,
we
can
infer
the
reliability
of
the
vehicles
for
a
particular
company.
A
gain,
I'll
be
showing
off,
this
is
a
JMP statistical
discovery
conference,
so
I'll
be
showing
off
some
tools
in
JMP
that
allow
us
to
get
that
information.
Wi
th
that,
let's
quickly
talk
about
what
data
are
we
looking
at?
W
hat
are
in
these
reports?
F
irst,
these
are
annual
reports,
and
I'm
going
to
be
focusing
on
a
single
company,
those
coming
from
Waymo,
which
is
formerly
this
Google
self-driving
car
project.
T
hese
are
reports
submitted
by
Waymo
to
the
California
DMV.
Now,
I've
been
talking
about
a
disengagement
event.
What
do
I
mean
by
that?
Well,
there's
two
types
of
testing.
The
first
involves there
being
a
driver
in
the
seat,
the
vehicle
actually
can't
operate
without
a
driver
in
the
seat.
But
at
some
point,
they
can
turn
on
autonomous
mode,
and
if
anything
happens
where
the
driver
has
to
take
over,
that
was
not
planned
as
part
of
the
testing,
that's
a
disengagement
event.
The
other
type
of
testing,
of
course,
is
a
completely
driverless
vehicle.
We're
not
going
to
focus
on
those.
We're
just
going
to
focus
on
the
driver
necessary
systems.
Now,
these
annual
reports
go
all
the
way
back
to
about
2015,
even
containing
information
back
to
2014,
which
is
about
when
this
whole
California
DMV
program
started.
Waymo
was
actually
one
of
the
first
companies
to
get
involved
in
this.
T
hese
reports
contain,
first
of
all,
they
contain
data
typically
from
December
of
the
previous
year,
to
November
of
the
current
year
of
the
report,
the
exception
being
the
2015,
which
actually
goes
back
a
bit
further
into
2014.
F
or
example,
the
2016
report
would
have
data
from
December
of
2015,
all
the
way
up
to
November
of
2016.
T
here's
two
general
types
of
data
present.
The
first
is
going
to
tell
you
all
about
the
disengagement
event.
What
was
the
cause?
Where
did
it
happen?
S
ometimes it
might
give
you
the
day
and
the
VIN
of
the
vehicle
involved
in
the
incident.
The
other
type
of
report
is,
how
many
miles
were
driven
in
autonomous
mode
for
a
vehicle,
for
a
particular
month.
Now,
you
may
have
noticed
that I
said,
in
the
disengagement
event
report,
sometimes
we
know
the
date
and
the
event.
Yes,
that's
right.
It's
not
very
consistent
across
the
entire
time.
In
fact,
that
information
actually
wasn't
available
until
2018.
Prior
to
that,
we
only
know
what
the
event
was
and
how
many
happened
within
a
month.
Starting
in
2018,
they
started
providing
that
information,
most
likely
because
the
DMV
encouraged
them
to
do
so.
W
hat
it
means
then
is
that
you
actually
have
two
levels
of
resolution
with
our
data.
T
he
first
is
at
a
month
level,
and
for
that
we
can
go
all
the
way
back
to
when
testing
began.
I'm
going
to
be
looking
at
that
type
of
data
using
our
reliability
growth
platform.
T
he
other
type
of
data
is
actually
daily
and
at
the
VIN
level,
so
individual
vehicle
level.
I'll
be
using
the
recurrent
events
platform
to
analyze
that.
Before
I continue
further,
let
me
show
you
where
those
are.
You'll
find
them
under
the
Analyze
Menu,
go
down
to
Reliability
and
Survival.
You'll
notice
a
ton
of
platforms
here.
The
one
I'll
be
highlighting
are
Recurrence
Analysis.
Th
is
is,
we
got
individuals,
in
this
case,
vehicles,
that
can
experience
a
recurring
event
over
time,
in
this
case,
a
disengagement
event,
so
we'll
be
using
that
for
the
VIN
level,
and
then
the
reliability
growth
I'll
be
using
to
analyze
the
monthly
aggregate
data.
The
same
idea,
although
reliability
growth
is
more
for
if
I
have
a
big
system
that
could
encounter m
ultiple
events,
sometimes
they
have
to
be
repaired
as
part
of
the
event,
so
they're
experiencing
it
over
time.
It
might
go
through
different
phases
in,
so
the
underlying
process
or
software
might
be
evolving
over
time.
That's
something
that
the
reliability
growth
platform
is
intended
to
handle.
Now
this
could
also
handle
the
VIN
information
as
well.
It's
fine
with
that.
I'm
just
picking
these
two
because
I
want
to
show
off
the
breadth
of
the
stuff
that
we
have
here
at
JMP.
While
I'm
doing
that,
I
also
want
to
show
you
a
bit
of
how
we
use
some
JMP
tools
to
compile
the
data.
Obviously,
there's
a
lot
of
reports,
so
how
do
we
compile
all
that
together
into
a
single
report?
I'll
touch
briefly
on
some
of
the
tools
that
we
used
in
JMP
to
help
with
that.
L
et's
dive
right
into
that.
F
irst
I'll
talk
about
reading
in
some
of
the
earlier
reports.
Y
ou've
already
noticed
there's
a
bit
of
inconsistency
in
what's
reported.
There's
also
a
bit
of
inconsistency
in
the
format
of
the
reporting,
so
some
of
the
very
early
reports
were
in
the
forms
of
PDF.
Let
me
show
you
an
example
using
the
2017
report.
W
e
have
a
nice
report
here
from
Waymo.
It's
very
well
formatted.
In
fact,
this
is
probably
one
of
the
better,
nicer
reports
that
we've
seen
amongst
the
companies
that
were
participating
at
this
time.
A
bunch
of
tables
in
here,
The
ones
we
really
want
are
the
ones
here
at
the
end
in
the
appendices.
This
is
an
events
report.
Notice
days
a
bit
of
a
misnomer.
It's
actually
the
month
where
it
happened,
some
information
about
the
operation,
what
caused
it.
We
definitely
want
this
information.
This
is
a
table
that
we
want.
We
also
want
a
table
here
at
the
end
which
shows
the
autonomous
miles
for
each
event,
in
this
case
the
last
four.
Don't
even
get
the
complete
Vin
for
each
vehicle
for
each
month.
Now,
of
course
it's
in
the
PDF,
so
we
really
don't
want
to
have
to
go
in
and
manually
type
these
in.
It
would
be
nice
if
JMP
had
a
way
to
read
in
tables
from
a
PDF.
Well,
you're
in
luck
because
I
can
go
here
and
do
open.
There's
my
pdf
I've
marked
it
to,
Select
all
files,
not
just
JMP
files.
There's
my
report,
I'm
going
to
click
Open
and
we
have
our
fancy PDF
reader.
Now,
what
it's
gone
through
is
it's
trying
to
identify
everything
that's
a
table
in
that
report
and
it's
done
a
pretty
good
job.
Scroll
through
here.
You
can
see
captured
most
of
those
tables.
In
fact,
probably
too
many
tables
don't
need
all
of
these.
You
can
scroll
through
them
individually
using
this
dropdown
menu
and
you
can
see
it's
labeled
by
page
and
the
table
number
on
that
page.
I
don't
need
all
of
these
tables
as
I
mentioned
before,
I
just
want
the
ones
that
are
in
the
appendices.
That's
easy.
I'll
go
to
the
Red
Triangle
menu
that
suddenly
appeared
on
the
page
and
say,
Just
ignore
all
the
tables
on
this
page.
I'll
scroll
down
and
do
the
same
thing
for
this
page.
Okay,
now
I've
got
the
tables
I'm
interested
in.
We
still
need
a
bit
of
tweaking
to
do
because
as
you'll
see,
they're
essentially
each
table
on
each
page
is
its
own
entity.
So
these
are
actually
all
the
same
table.
How
do
I
tell
JMP
to
do
that?
Well,
now
I
can
go
to
the
Red
Triangle
menu
on
the
table.
I'm
going
to
scroll
down
here
to
number
of
rows
to
use
this
header
and
tell
it
to
use
none.
Now
what
this
means
is
if
there's
a
table
prior
to
it
that's
telling
JMP
that,
hey,
that's
actually
part
of
the
previous
table.
That's
why
there's
no
header
on
this.
I'm
going
to
click
zero.
You'll
notice
that
table,
there's
one
less.
You've
also
noticed
it's
added
two
columns
in
this
case.
What
page
is
that
table
occur
and
what
table
on
that
page.
If
you
wanted
that
information,
JMP
automatically
provides
it.
In
this
case,
I
don't
really
need
it,
but
I
can
delete
it
later
after
I'm
done
reading
these
then.
We
also
notice
we
missed
a
bit
of
that
table
probably
because
the
formatting
change
notice
it
was
able
to
pretty
easily
find
what
the
roads
were
for
this
table,
but
for
this
one,
not
so
much.
Mainly
because
of
this
guy.
Some
we
just
didn't
put
it
just
right,
so
it
didn't
fit
the
pattern.
That's
okay,
I'm
going
to
click,
I'm
going
to
drag
around
and
that
tells
JMP,
"Hey,
there's
you
a
new
table",
and
I
can
go
in
right
now
thinks
it's
a
zone
table.
I
can
just
tell
it
no,
it's
part
of
the
previous
one.
There
we
are.
I'm
going
to
scroll
through
and
yeah,
it
missed
some
of
this
information.
But
again,
notice
it's
trying
to
maintain
that
formatting,
but
because
of
the
way
it
was
input,
it
missed
that
last
row.
That's
okay
I
can
input
that
information
later.
It's
not
a
big
deal.
If
we
go
to
this
next
page
in
this
case
JMPs
done
the
opposite.
It's
actually
concatenated
these
two
together,
which
makes
sense.
It
sees
these
are
pretty
close
together.
Maybe
they
are
the
same
table.
Unfortunately,
they
are
not.
As
you
can
see,
they're
two
distinct
tables
here,
whereas
here
they're
concatenated
together.
I
need
to
tell
JMP.
No,
that's
actually
a
different
table.
That's
pretty
easy.
Just
say
how
many
rows.
No,
there's
actually
is
a
header
row,
and
that
tells
JMP,
that's
a
different
table.
Now
it's
created
a
different
table.
I
would
have
to
repeat
that
for
a
lot
of
these.
Unfortunately,
the
way
they've
input
this,
I'm
going
to
have
to
do
a
bit
of
work
later
on
to
get
them
in
the
right
format.
But
that's
okay.
There
are
other
tools
and
JMP
to
do
that.
For
right
now
for
the
sake
of
time
I'm
not
going
to
go
into
that.
I
just
want
to
show
you
how
we
could
read
it
in
from
a
PDF.
Now,
thankfully,
starting
in
about
2018,
they
started
using
Excel
for
their
reports.
It
was
a
lot
easier
to
just
copy
and
paste
directly
into
JMP.
Now
that
we've
got
the
raw
data
tables
in,
I'm
not
done
yet
because
I
would
like
to
reformat
some
of
these
data
tables
to
get
them
into
a
nice
common
formatting
that
can
then
use
to
concatenate
them
all
together
into
a
single
table.
Now
I'll
show
you
how
I
did
that,
and
it's
actually
quite
a
lot
of
things
that
I
want
to
do.
Here's
an
example
of
the
Waymo
2022
report.
We
in
this
case,
we
have
the
date
of
the
incident,
the
VIN
number
of
the
vehicle
involved.
This
is
just
extra
information
that
I
actually
don't
want
to
keep.
It's
just
saying,
can
I
drive
by
itself?
Is
it
completely
driverless?
No.
Yes,
sir.
Of
course
there's
a
driver
present.
Who
initiated
it.
I'm
not
going
to
use
that
information.
I'm
going
to
keep
the
location
and
the
description.
This
is
essentially
the
cause.
I
will
probably
rename
some
of
those
columns.
I
also
need
to
summarize
over
this
table,
because
from
this
I'd
like
to
create
an
aggregate
events
so
I
can
use
it
with
the
monthly
aggregate
data.
I would
need
to
aggregate
summarize
over
the
VIN
numbers
over
the
dates
by
month.
Do
all
that,
so if
you
bear
with
me,
I've
a
lot
of
about
five
minutes
or
so
to
do
that.
I
know
it's
a
bit
dangerous
to
kind
of
do
that
live,
but
just
bear
with
me.
First
thing
I
need
to
do
is
I
need
to
open
this
guy
and
then
I
need
to
click
this
button.
Okay.
Give
me.
Let
me
take
a
little
break
here.
That
was
a
lot.
Thank
you
for
bearing
with
me
on
that.
As
you
can
see,
thanks
to
something
new
called
the
JMP
workflow,
I
was
able
to
accomplish
all
of
that
in
a
fraction
of
a
second.
Well,
maybe
a
little
over
a
second,
but
still
a
lot
faster.
I've
got
a
data
table
here
aggregated
by
month.
I got
the
location,
the
cores,
how
many
events
happened
in
that?
I
can
relabel
some
of
that
and
I've
got
my
individual
data
table
that
has
the
date,
the
VIN
and
the
actual
individual
events.
I've
got
that
set
up.
Now
I
can
save
those
in
different
locations
to
be
concatenated
later.
What
is
this
new
workflow?
Well,
let
me
open
up
a
new
one
for
you
just
so
you
can
see
it
if
can
get
there.
There
we
go.
This
is
what
you
would
start
off
with.
What
you
do
is
you
click
this
red
record
button
and
on
an
individual
data
table
you
would
then
it
would
then
record
the
actions
you're
taking
on
that
data
table.
This
is
very
useful
if
you
have
an
this
like
I
have
here,
I
have
a
lot
of
reports.
I'm
going
to
be
doing
the
same
thing
for
each
report.
Then
having
to
instead
repeat
all
those
actions
for
each
of
those
tables,
I
just
record
it
for
one
table
and
then
save
it
as
a
workflow.
Then
when
I
ran
it,
what
it
would
do
is
it
would
open
up
a
window
saying,
"Hey,
couldn't
find
that
original
data
table.
Can
you
show
me
what
the
new
one
is?"
Select
it
and
it
runs
it.
Now,
in
this
case,
it
didn't
do
it
here
because
I
already
had
it
primed
for
this
data
table.
But
typically
if
the
data
table
has
changed
in
the
name,
so
long
as
it
keeps
the
same
formatting,
that's
the
only
thing
you
have
to
worry
about.
If
a
column
did
change
a
name,
that's
okay.
JMP
also
knows
to
ask,
"Hey,
couldn't
find
that
column.
Can
you
give
me
a
replacement?"
This
is
very
helpful.
Let's
say
you
have
a
daily
report
that
you
have
to
collect
data
on
so
you
would
get
a
data
table
of
that
report.
Maybe
there's
a
couple
cleaning
things
you
need
to
do
on
that
data
table.
Save
a
workflow,
open
that
data
table,
click
Run,
select
the
right
data
table
and
you're
done.
This
is
a
great
time
saver.
This
really
helped
save
us
a
lot
of
time
when
we
were
compiling
these
data
together.
That
was
something
that
was
just
recently
introduced
in
JMP
17.
I'm
sure
you're
going
to
love
that.
Let
me
exit
out
of
all
of
these.
I've
already
got
those
tables
done,
and
want
to
make
sure
we're
going
to
go
on
to
the
analysis
because
want
to
make
sure
we
have
time
to
do
that.
Let's
start
with
the
monthly
aggregate
level.
Here
I've
compiled
it
across
all
the
time
periods
up
until
the
most
recent,
that
being
2022.
Let's
quickly
walk
through
what
I've
got
here.
I've
got
date
in
this
case,
it's
the
month.
How
many
vehicles
are
actively
being
tested
in
that
period?
How
many
total
autonomous
miles
were
driven
in
that
month?
This
is
how
many
different
event
types
happened.
This
is
the
total
number
of
disengagement
events
that
occurred.
Let's
do
a
quick
visual
look
at
this.
JMP
is
all
about
graphs
and
visual
exploration.
Let's
look
at
that.
Here
I've
got
plotted
the
cumulative
number
of
events
over
time
and
the
individual
events
and
we
notice
is
an
interesting
little
pattern
here.
There
seem
to
be
there's
clear
spikes
and
then
drops
and
it
seems
to
happen
about
4
or
5
times
where
we
see
this
spikes.
You
can
also
see
that
with
the
cumulative
data.
Ideally
what
you
would
like
with
this
type
of
data,
there'd
be
a
bit
of
an
increase
at
the
beginning
and
then
it's
going
to
flatten
out.
Ideally,
it
becomes
completely
flat.
If
you
do
that,
you
have
achieved
the
perfect
autonomous
vehicle.
You
have
no
more
disengagement
incidents
to
worry
about.
But
of
course
that's
never
going
to
happen.
You
just
want
it
to
be
as
flat
as
possible.
What
we're
seeing,
of
course,
is
a
bit
of
a
rise
or
a
burn
in
period,
if
you
will,
and
then
it's
starting
to
flat
out.
But
we
have
that
repeated
a
couple
of
times
most
significantly
here.
That's
a
pretty
big
jump
there.
Of
course,
you
can
see
that
correlating
here.
We've
got
our
burn
in
period.
Then
as
we
drop
in
the
number
of
incidents,
our
curve
up
top
is
going
to
flatten
out.
This
is
going
to
be
the
key
piece
of
information
we are going
to
be
looking
at
in
our
reliability
platforms.
We've
got
that
information
there.
Now
I'll
go
ahead
and
quickly
address
one
question
that
might
come
up.
Maybe
you
might
think,
well,
maybe
the
more
you
drive
it,
the
more
incidents
you
might
have.
Maybe
we
should
try
and
account
for
that
and
I
have.
Here's
another
graph
where
I've
looked
at
the
correlation
between,
in
this
case,
the
log
of
the
mileage
and
the
log
of
the
total
disengagement
events.
There
is
a
bit
of
a
trend.
It's
a
very
small
one.
In
fact,
the
R²
is
not
good
at
all.
There's
a
lot
of
spread
in
the
data.
Here's
actually
the
prediction
equation
with
its
individual
prediction
boundaries,
there's
a
lot
going
on.
There's
plenty
of
room
for
a
flat
line
there.
This
is
just
saying
that
there
might
be
something
like
that
happening,
but
it's
clearly
not
the
driving
force.
Yes,
if
you
drive
it
a
little
bit
more,
you
might
experience
a
few
more
incidents
that
make
sense.
But
it's
clearly
not
a
big
driving
force.
Something
else
is
clearly
going
on.
Hopefully
that
helps
address
a
side
question
that
might
come
up.
Well,
let's
get
into
the
actual
reliability
analysis.
I'm
going
to
run
this
script
here,
and
this
is
going
to
run
that
reliability
growth
platform.
Don't
worry.
After
I've
run
it,
I'm
going
to
click
here
under
the
Red
Triangle
redo
relaunch.
This
brings
up
the
initial
launch.
This
is
what
the
initial
launch
window
looks
like.
You
have
four
different
ways
of
entering
the
data.
The
first
is
a
time
to
event.
If
you
had
recorded
here's
how
many
days
or
months
up
until
an
event
happened,
you
would
use
this
one.
Then
you
could
of
course,
maybe
multiple
happened
at
that
time.
You
could
put
that
there.
We
have
the
dates,
which
is
what
I'll
be
using
because
we
instead
of
an
actual
timed
event
we
just
had
saying
that
in
this
month
this
many
events
happen.
We
have
a
timestamp
and
an
event
count
for
that.
You
also
have
it
if
you
have
individual
information,
maybe
we're
doing
concurrent
testing
or
testing
in
parallel.
You
can
do
that
information
here.
Notice
there's
a
spot
for
the
system I
D.
If
I
were
looking
at
the
individual
event
information,
this
is
where
I
would
go.
But
for
right
now,
we're
going
to
stick
with
the
dates.
I'll
revisit
the
phase
stuff
in
a
moment.
You'll
notice
it's
plotted
that
cumulative
curve,
which
we
saw
earlier.
There
it
is
right
there.
Now
the
metric
of
interest
here
is
what's
called
mean
time
between
failures.
It's
clearly
oriented
towards
reliability
data
where
we're
looking
at
failures.
Think
of
this
as
what's
the
average
month
between
incidents,
disengagement
incidents.
There's
a
bit
of
a
non-parametric
measure
here
that's
just
sort
of
a
moving
window,
looking
at
it
over
time.
But
you
can
get
a
sense
of
how
good
we
get,
maybe
how
poor
we
get.
I'm
actually
going
to
fit
a
model.
The
model
I'll
be
fitting
is
here
under
fit
model.
This
was
a
bit
of
a
mouthful.
It's
a
piecewise
viable
in
change
point
detection.
Let
me
break
that
down
for
you.
The
typical
model
used
in
modeling
this
type
of
account
type
data
is
what's
called
a
Poisson
process.
What
that's
essentially
says
is
that
the
rate
at
which
incidents
occur
in
this
case
rate
per
month
is
constant
over
time.
That
would
be
the
equivalent
of
a
straight
line
here
from
a
mathematical
perspective,
that's
a
nice
model.
It's
very
easy
to
analyze
,
from
a
practical
perspective
that
is
a
horrible
model
because
we
never
get
to
improve.
Our
rate
of
incident
is
always
the
same.
We
don't
want
a
constant
rate.
We
want
the
rate
to
go
to
zero,
preferably.
That's
what
I
mean
by
that
flattening
of
the
curve.
What
we
want
is
what's
called
a
Non-homogeneous
Poisson
process,
fancy
term
meaning
the
rate
changes
over
time.
The
weibel
part
says
how
exactly
does
it
change
over
time?
It
consists
of
two
parts.
There's
an
initial
rate
lambda
Think
of
that
as
like
a
constant
out
front
and
multiply
that
by
time
raised
to
some
power
beta.
That
value
beta
is
going
to
tell
you
how
the
rate
changes.
If
beta
is
greater
than
one,
my
rate
is
increasing
over
time.
That's
bad,
we
don't
want
that.
That
rate
were
zero.
We're
at
a
Poisson
process.
It's
constant,
we
also
don't
want
that.
What
we're
looking
for
is
a
value
of
beta
that's
between
0
and
1
saying,
you
know,
okay,
it'll
increase
a
little
bit,
but
then
it's
going
to
start
to
die
off,
it's
going
to
slow
down
getting
us
that
leveling
off
that
we
want.
When
we
estimate
these,
we're
looking
at
a
value
between
0
to
1.
We're
also
looking
visually
for
that
flattening
off.
For
the
meantime,
between
failures,
what
we
want
is
that
to
get
very
large,
ideally
it'd
be
infinite,
meaning
it'll
never
happen,
but
of
course
that
can't
happen.
We
just
want
very
large
mean
times
between
failures.
I'm
going
to
fit
that
model
here.
I'm
going
to
use
a
change
point
because
that's
saying
that
at
some
point
maybe
that
value
of
beta
in
my
model
is
going
to
change
significantly,
mainly
because
of
this
guy
here.
I'm
going
to
run
that
because
it
looks
so
different.
Yes,
according
to
this,
there
is
a
change
point
right
there,
a
pretty
significant
one.
If
you
look
graphically,
that's
not
a
great
fit,
but
it's
not
a
really
bad
fit
either.
But
it
is
detecting
that
something
significant
happened
in
2021.
If
we
go
to
this
plot,
this
is
showing
you
how
does
that
mean
time
between
failure
changed
over
time
and
it's
doing
it
for
each
in
this
case
phase.
This
is
sort
of
the
empirical
phase
based
on
this
change
point.
In
the
first
phase,
it
says,
well,
we
got
to
about
as
good
as
0.15
months.
That's
roughly
five
days
between
incidents,
not
too
bad.
Roughly
a
week.
Again,
this
is
across
a
fleet
of
vehicles.
For
any
individual
vehicle,
it
could
be
longer
or
shorter.
But
for
the
fleet,
essentially
the
software
itself,
about
a
week
between
disengagement
incidents.
After
2021,
it
drops
significantly
to
about
5%.
That's
almost
a
day
and
a
half.
That's
quite
a
drop.
We
can
look
at
the
parameter
estimates.
There's
my
data,
that's
just
my
initial.
Notice
For
beta
one
it's
less
than
one,
we're
doing
great.
For
the
second
phase,
not
so
much.
In
fact
we
can
see
maybe
we
could
do
a
little
bit
better.
Notice
there
were
about
four
of
those,
so
maybe
I
can
invoke
some
sort
of
empirical
grouping
on
that,
which
is
what
I've
done
in
the
other
script.
It's
going
to
run
this.
All
I
did
here.
I'll
do
the
redo,
relaunch?
I
just
added
a
column
called
Empirical
Phase
that's
just
got
a
bunch
of
letters
in
it
that
just
distinguish
the
phases
from
each
other.
It's
a
categorical
factor.
That's
all
it
is,
and
that's
what
these
vertical
lines
represent.
This
is
a
combination
of
me
repeatedly
using
the
change
point.
It
only
detects
one
change
point.
You
can
do
a
trick
where
I
can
select
some
of
the
data,
hide
and
exclude
it,
and
then
do
it
again.
That's
a
way
for
it
to
drive
detect
different
change
points.
Some
of
this
is
also
just
me
eyeballing
it.
Full
confession.
Now
for
this
one,
the
model
all
fit
is
called
a
piecewise
Weeble.
The
same
thing
is
what
we
did
before.
But
instead
of
it
detecting
the
change
point,
it's
saying,
I've
already
got
those
in
for
you.
Each
time
you
see
one
of
these
changes,
these
different
phases,
that
value
of
beta
is
going
to
change.
It's
going
to
connect
them
all
together.
But
that
value
of
beta
might
change,
indicating
a
change
in
the
process
overall
based
on
these
phases.
This
is
a
much
better
fit.
It's
maybe
it's
a
bit
of
a
too
much
overfitting.
I
will
acknowledge
that.
But
if
we
look
at
this,
what
it's
essentially
saying
is
for
that
first
initial
period
of
changes
in
the
software,
were
you
about
got
almost
as
good
as
0.4
months.
That's
almost
12
days
between
incidents.
It's
still
saying
that
we
got
really
good
up
until
2021
and
then
we've
dropped.
Clearly
something
is
going
on
in
around
2021.
Something
significant
has
changed
in
the
underlying
software
across
the
fleet
of
vehicles
that's
dropped.
That's
increased
its
incidence
rate.
There's
those
parameter
estimates.
Again,
notice
I'm
going
to
ignore
that
one.
That's
a
burning
period.
A
lot
of
these
are
less
than
one
until
we
get
to
the
season
D's,
which
are
afterwards.
Clearly
something's
happening
because
it's
larger
than
one.
We're
already
we've
got
a
bit
of
a
story.
Something
clearly
happened
in
2021
to
change
the
incident
rate.
Maybe
when
we
look
at
the
individual
data,
we'll
figure
out
what's
going
on.
Let
me
clear
out
all
of
this
and
let's
move
over
to
the
individual
vehicles.
Quick
summary.
We
have
our
date.
The
actual
date
of
the
incident,
the
vehicle
identification
number,
the VIN
this
VIN
group,
it's
nothing
fancy.
It's
just
the
first
four
digits
of
the VIN.
You're
going
to
see
why
I've
created
a
group
in
a
moment.
I've
got
the
location,
the
cause.
I've
reduced
the
text
of
the
cause.
It
was
very
wordy.
I
just
reduced
it
to
a
basic
description.
We'll
cover
that
in
just
a
moment.
The
starting
and
ending
month.
I've
got
this
information
basically
telling
me
how
long
was
this
vehicle
under
test?
This
particular
vehicle,
it
was
basically
being
tested
in
autonomous
mode
on
and
off
from
December
of
2017
to
about
November,
maybe
October
of
2019.
We're
going
to
need
that
information.
I'm
going
to
first
look
at
some
basic
some
graphical
summary.
First
thing
I'm
going
to
do,
I'm
going
to
run
this.
This
is
just
going
to
summarize
over
the
causes.
Let
me
look
at
the
total
number
of
disengagement
events
for
each
vehicle
each
month.
Then
I'm
going
to
run
this
plot
and
you're
going
to
see
it's
going
to
look
interesting,
like
coral
under
the
sea,
waving
about
very
pretty.
A
lot
of
information
going
on.
But
the
basic
thing
I
want
to
tell
you
is
the
way
to
interpret
this
is
like
we
did
with
the
cumulative
events
graph.
That's
what
these
are.
You
want
to
see
a
bit
of
a
burn
in,
but
it
ultimately
flattening
off.
Now
I've
got
a
colored
by
individual
then,
which
is
why
you
don't
see
the
labels.
There'd
be
way
too
many.
I'm
going
to
color
by
a
different
variable
this
time.
We're
going
to
color
by
the
VIN
group.
Come
here
under
color.
Again,
that's
just
the
first
four
digits
of
the
VIN
number,
which
clued
me
in
because
let
me
throw
in
the
legend
so
you
can
actually
see
it.
Show
legend.
I'm
done.
Notice
there
was
a
significant
change
in
the
VIN
number
indicating
maybe
a
significant
change
in
the
vehicle.
Notice
when
the
change
occurred
right
around
2021.
Very
interesting.
You
can
also
see
a
general
trend
in
the
behavior
for
these
early
vehicles.
Now,
some
of
these
probably
actually
started
earlier
than
this.
There's
a
bit
of
censoring
involved,
but
already
they're
pretty
flat.
We've
got
a
bit
of
burn
in
and
then
some
flattening
off.
That's
great,
that's
what
we
want
to
see.
These
guys,
there's
almost
a
straight
line
up.
We've
got
a
lot
of
incidents
happening
in
a
very
short
period
of
time.
Now,
later
on
here,
later
in
2022,
there
are
starting
to
lean
off
a
little
bit.
We've
got
some
that
did
start
and
they're
starting
to
be
pretty
shallow
curve.
That's
good.
But
clearly
something's
happened
here.
This
seems
to
indicate
that
a
different
vehicle
and
perhaps
a
different
type
of
software
was
introduced
about
2021.
It's
having
some
issues.
It's
struggling
a
bit,
or
at
least
initially
did.
About
here
it's
performing
about
as
well
as
we
saw
early
on.
They've
they've
done
better
in
fixing
it.
They
introduced
something
new
and
had
a
bunch
of
bugs.
That's
what
happens
when
you
introduce
new
things.
Let
me
exit
out
of
this.
Let's
look
at
maybe
something
about
a
different
type
of
cause.
Now,
remember,
in
the
previous
video,
we
didn't
really
look
at
the
cause
because
it
was
aggregated
across
all
the
vehicles.
But
let's
look
into
that
real
quick.
Maybe
there
might
be
something
going
on
there.
Before
we
actually
get
into
the
recurrence
data.
Here
I've
just
done
a
basic
bar
chart
of
the
causes,
and
a
lot
of
them
are
things
you
might
expect
to
be
a
cause.
An
unwanted
maneuver
equals
doing
something
you
don't
want
it
to
do,
a
perception
discrepancy.
The
vehicle
didn't
see
something
it
should
have
or
thought
it
saw
something
that
wasn't
there.
Then
there's,
of
course,
software,
a
general
catchall
software
discrepancy.
Something
happened
with
the
software.
There
are
some
interesting
causes.
For
example,
other
people
being
jerks
on
the
road.
A
reckless
road
user
car
didn't
know
what
to
do.
Adverse
weather,
incorrect
behavior
prediction,
hardware
discrepancy.
These
are
the
basic
causes.
Now,
I'm
going
to
use
a
feature
called
a
Graph
Lit
to
investigate
things
further.
I'm
going
to
right
click
inside
this
bar.
I'm
going
to
hover
label,
I'm
going
to
click
Bar.
What
it's
done
is
it's
added
a
little
bar
chart
Graph Lit.
When
I
leave
my
bars
there
with
a
hover,
here's
my
little
Graph Lit.
Now,
by
default,
it's
going
to
use
the
first
categorical
factor
it
sees,
which
was
the
variant
number.
I
don't
want
that.
I
actually
want
the
date.
I'm
going
to
open
the
control
panel
here.
I'm
going
to
put
the
date
here
and
replace
it.
I'm
going
to
right
click
in
the
date.
Because
it's
a
date.
This
was
recently
introduced.
Think
it
was
either
16
or
17.
I
can
actually
bend
by
a
particular
type
of
date,
in
this
case I
want
to
bend
by
a
month?
Perfect.
I
also
would
like
to
color
by
the
VIN group.
Just
so
that
we
get
a
clear
comparison
of
the
different
types
of
groups.
Notice
there's
a
bit
of
purple
here.
That
means
there
is
some
overlap,
which
makes
sense.
There
was
a
bit
of
overlap
from
end
of
2020
to
2021
with
a
new
groups,
but
now
we
can
clearly
see,
compare
visually
the
groups
across
the
types
of
incidents
just
visually.
I'm
going
to
come
here,
I'm
going
to
go
back
down
to
save
script
to
a
clipboard.
I'm
going
to
exit
out
of
this
because
right
now,
if
I
hover,
it's
no
longer
there,
but
I
can
right
click
go
to
hover
label
and
paste
Graph Lit.
Now
it's
going
to
show
me
the
graph
for
each
of
them.
In
fact,
I'm
not
going
to
click
and
zoom
in
because
we
can
see
what's
happening.
There's
not
too
much
there's
this
is
the
unwanted
maneuver.
There
might
have
been
a
bit
more
unwanted
maneuvers,
but
that's
more
of
a
general
spike
in
overall
terms,
not
much
difference,
I
would
say,
for
the
perception
discrepancy
pretty
much
on
par.
There's
a
bit
more
in
general
here,
but
I'd
still
say
comparable.
Sort
for
discrepancy.
Whoa,
let's
look
into
that.
Okay,
This
is
very
interesting.
Apparently
with
the
old
group,
there
was
hardly
any
software
discrepancies,
maybe
one.
Whereas
with
the
new
group,
there's
a
lot.
Interesting.
There
could
be
some
explanations.
It
could
be
that
the
way
the
category
was
defined
maybe
changed
over
time.
Maybe
there
were
other
things.
It
was
re
categorized
as
other
things
for
the
old
one
before
they
change
the
definition
for
this
new
group.
That's
a
very
valid
explanation.
It
could
also
be
taken
at
face
value.
Maybe
there
were
a
lot
of
other
different
causes
with
the
old
one.
Maybe
if
we
look,
we're
able
to
look
back
early
on
at
the
individual
vehicle
information
for
the
earlier
data,
maybe
we
would
have
seen
a
spike
in
the
software
discrepancies.
Because
this
is
later
in
that
series
testing,
maybe
we
did
those
out
a
little
bit.
We're
just
seeing
these
early
bugs
happening
with
the
new
vehicles.
Let's
actually
now
go
into
the
recurrence
event
analysis
now
for
current
events.
That's
just
saying
have
an
individual
object
that
can
experience
an
event
repeatedly,
in
this
case
a
disengagement
event.
Over
time.
I'm
going
to
run
the
script
and
then
do
the
redo
relaunch
so
you
can
see
what
I
did.
Do.
I'm
going
to
do
relaunch
by
group.
In
this
case,
here's
what
you
would
get
if
you
opened
it
up.
In
this
case,
you
have
an
age
or
event
timestamp.
When
did
it
happen?
Have
a
timestamp.
The
date.
What's
the
system?
That's
the
VIN.
What's
the
cause?
Well,
hey,
great
to
have
a
column
called
cause
it's
going
to
basically
break
it
down
by
the
different
causes.
You
can
look
at
these
different
causes.
The
timestamp.
When
did
I
start
testing
on
this
thing?
Basically,
when
does
it
undergo
start?
When
did
I
stop
looking
at
it?
What's
the
timestamp?
I
put
the
VIN
group
in
by,
I
could
have
put
it
in
grouping,
but
there
is
a
bit
of
an
issue
where
if
there's
not
a
lot
of
data
in
the
causes
for
both
of
them,
it's
not
going
to
be
able
to
compare
them.
For
right
now,
I'm
just
going
to
use
a
by
group, a VIN
by
group.
We
can
still
do
a
sort
of
visual
comparison.
For
the
scaling
you
can
select
that
I've
done
today
because
that's
the
level
of
resolution
we
have.
It'll
do
it
in
a
day
just
for
interpretation,
ease
of
that.
Okay,
so
we're
looking
at
this
first
group.
This
is
the
early
group
two
for
our
group.
Notice
it's
broken
it
up
by
all
the
different
causes
and
they're
pretty
similar.
The
hardware
discrepancy
seems
like
the
biggest
one,
if
I'm
reading
that
correctly.
Yes,
that's
hardware.
Notice
this
is
what's
called
a
mean
cost
function.
It's
similar
in
interpretation
to
that
cumulative
function.
You're
looking
for
the
same
behavior,
a
bit
of
a
rise,
but
you
ultimately
want
it
to
flatten
out.
This
is
saying
what's
the
average
sort
of
incidence
rate
for
a
single
vehicle
by
this
age?
It's
a
little
bit
harder
to
interpret
than
just
what's
the
mean
total
count
because
it's
accumulating
over
time.
Just
at
a
high
level
think,
I
want
to
rise
but
also
want
it
to
flatten
out
at
some
point.
So that
I'm
getting
no
more
incidence
for
that
vehicle.
That's
what
we're
seeing
with
a
lot
of
these.
There's
a
bit
of
a
rise
and
it's
starting
to
shallow
out.
If
you
want
detailed
information,
you
can
look
under
each
one.
We
can
do
some
fit
models,
I'm
not
going
to
at
this
point,
because
we're
starting
near
the
end,
but
it
would
be
similar
types
of
models
that
you
would
fit
here.
Let's
look
at
the
SAD
Group,
if
you
will.
There's
clearly
a
difference
here.
Notice
this
big
spike
and
again,
that
is
the
software
discrepancy.
What
we're
seeing
is
we've
essentially
got
a
new
vehicle.
You
can
you
can
clearly
see
if
we
compare
that
is
starting
out
later.
We
had
a
couple,
maybe
some
early
on
while
the
other
vehicles
being
tested.
Makes
sense.
You
got
to
sort
of
pilot
group
being
tested.
Then
when
they
did
full
scale
testing,
the
software
started
having
some
issues.
Now
it's
starting
to
get
better.
We're
seeing
that
level
out.
We
saw
that
in
the
previous
data.
There's
clearly
a
difference,
but
they
are
doing
better.
These
two,
the
unwanted
maneuver,
the
perception
discrepancy,
I
would
say
visually
they're
on
par
with
what
I
see
here.
Clearly
the
software
is
the
big
issue
here.
What's
the
big
takeaway
here?
What
would
we
say
about
this?
Well,
it
seems
that
we've
got
two
different
series
of
vehicles.
We've
got
one
that
they
were
working
on
for
a
while.
Then
in
2021,
they've
introduced
a
new
series
of
vehicles.
For
right
now,
especially
in
2021,
there
is
a
bit
of
issues
with
the
software
going
on.
A
lot
of
software
discrepancy
issues
causing
incidents.
They
seem
to
be
improving
that
now.
That
makes
sense.
You've
introduced
something,
there's
going
to
be
bugs.
Just
give
it
some
time.
All
the
other
ones
seem
comparable
with
what
was
going
before.
Overall,
I've
got
a
pretty
good
feeling
about
this
new
series.
Give
it
some
time
to
burn
in.
Well,
I
hope
I've
given
you
a
flavor
of
all
the
things
that
JMP
can
do,
especially
in
the
reliability
platform
as
well
as
in
preparing
some
of
your
data.
If
you
have
any
questions
about
this,
this
journal
will
be
available
where
you
can
find
this
talks.
Some
of
the
data
tables
will
be
available
there
as
well.
You
get
to
play
with
all
of
this.
There
are
also
some
other
graphs
didn't
really
go
into
some
other
analyzes,
so
feel
free
to
explore.
That's
what
we
do
here
to
JMP
statistical
discovery.
If
you
have
any
questions
about
it,
I'll
be
in
some
of
the
Meet
the
Expert
sessions
so
you
can
find
me
there.
I'd
love
to
chat
with
you
about
this
or
anything
else
you
might
have
about
designing
experiments
or
reliability.
Thank
you
for
your
time.
