Hello,
my
name
is
Caleb
King.
I'm
a
developer
in
the
DoE
and
reliability
group
at
JMP.
Today
I
figured
I'd
showcase
what
I
think
is
a
bit
of
an
overlooked
platform
in
the
reliability
suite
of
analysis
tools,
and
that's
the
reliability
growth
platform.
I
thought
I'd
do
that
in
the
context
of
something
that's
become
pretty
popular
nowadays,
and
that's
autonomous
vehicles.
They're
fast
becoming
a
reality,
not
so
much
science
fiction
anymore.
We
have
a
lot
of
companies
working
on
extensive
testing
of
these
vehicles.
It's
nice
to
test
these
vehicles
on
maybe
a
nice
track
at
your
lab
or
something
like
that.
But
nothing
beats
real
actual
road
testing,
which
is
why
early
in
the
2010s,
the
state
of
California's
Department
of
Motor
Vehicles
actually
put
together
a
testing
program
that
allowed
these
companies
to
test
their
vehicles
on
roads
within
the
state.
Now,
as
part
of
that
agreement,
each
company
was
required
to
submit
an
annual
report
which
would
detail
out
any
type
of
disengagement
incidents.
Or
heaven
forbid,
any
crashes
that
happened
involving
their
autonomous
vehicles.
Those
had
to
be
reported
to
the
Department
of
Motor
Vehicles,
the
DMV.
Now,
one
benefit
of
the
DMV
being
a
federal
institution
is
that
these
reports
are
actually
available
upon
request.
In
fact,
we
can
go
there
right
now
to
the
site
and
you'll
see
that
you
can
at
least
access
the
most
recent
reports.
We
have
the
2021
reports.
They're
still
compiling
the
2022.
You
could
also,
if
you
want,
email.
If
you
wanted
some
previous
ones,
I
did
that
with
a
brief
justification
of
what
I
was
doing.
They
were
pretty
quick
to
respond.
Now,
we
have
different
types
of
reports
and
different
types
of
testing.
We're
we're
focusing
on
testing
where
there
is
a
driver
in
the
vehicle
and
the
driver
can
take
over
as
necessary.
This
isn't
a
fully
autonomous
vehicle.
You
do
have
to
be
in
the
driver's
seat
to
do
this.
We're
using
these
disengagement
events
as
a
proxy
for
assessing
the
reliability
of
the
vehicles.
Obviously,
we
don't
have
access
to
the
software
in
these
vehicles.
If
you
worked
at
those
companies,
you
could
probably
have
more
information.
We
obviously
don't.
But
they're
a
proxy
because
if
you
want
our
vehicle
to
be
reliable,
that
means
it
needs
to
be
operating
as
you
intend
within
the
environment.
Any
time
you
have
to
take
over
the
AI
for
some
reason,
that
could
be
a
sign,
"I
t's
not
exactly
operating
as
I
intended."
We
can
use
it
as
a
bit
of
a
proxy.
Again,
it's
not
the
best
approximation,
but
it's
still
pretty
good.
Of
course,
I'm
not
the
first
one
to
think
of
this.
This
is
actually
an
informal
extension
of
some
work
I've
done
recently
with
my
advisor,
Yili
Hong,
and
a
bunch
of
other
co-authors
where
we
actually
looked
at
this
type
of
data
from
a
recurrent
events
perspective.
I'm
going
to
do
a
slightly
different
approach
here.
But
there
is
a
preprint
of
this
article
available
if
you
want
to
check
it
out
that
does
something
similar.
Let
me
go
in
and
describe
the
data
you
for
you
real
quick.
I'm
not
going
to
be
looking
at
every
company
doing
testing.
There's
so
many
out
there.
I'm
going
to
focus
on
one,
and
those
would
be
events
submitted
by
Waymo,
which
was
Google's
self
driving
car
project.
Now
they're
their
own
subsidiary
entity.
These
are
their
annual
reports.
Let
me
define
what
we
mean
by
disengagement
events.
I'm
in
the
driver's
seat
and
if
something's
happening,
I'm
in
autonomous
mode
and
I
need
to
take
over
and
take
over
driving.
That's
a
disengagement
event.
I
disengage
from
autonomous
mode.
That
could
be
for
any
reason.
They,
of
course,
need
to
report
what
that
reason
was.
We're
just
using
that
as
our
proxy
measure
here.
These
annual
reports
are
going
to
go
all
the
way
back
to
about
2015,
2014.
That's
when
Waymo started
participating
in
this
program.
The
2015
report
actually
contains
data
back
to
2014.
They
start
in
the
middle
there.
Each
report
essentially
covers
the
range
from
December
of
the
previous
year
to
November
of
the
current
year.
The
2016
report
would
contain
data
from
December
2015
up
to
November
of
2016.
That
way,
they
have
a
month
to
process
the
previous
year's
numbers.
There
are
two
primary
sources
of
data
we're
looking
at
in
each
report.
The
first
one
is
going
to
list
each
incident
that
occurred,
when
it
happened
that
could
be
as
detailed
as
day
and
time,
or
it
could
just
be
the
month.
Again,
there's
not
a
lot
of
good
consistency
across
years.
It's
something
we
ran
into.
But
they
at
least
give
some
indication
of
when
it
happened.
They
might
say
where
it
happened
and
they
can
describe
what
happened.
It
could
be
very
detailed
or
it
could
just
be
falling
into
a
particular
category
that
they
give.
Then
the
second
part
of
data
is
going
to
list
the
the
VIN
or
partial
VIN
of
the
vehicle,
so
the
vehicle
identification
number.
Something
to
identify
the
vehicle
and
how
many
autonomous
miles
that
vehicle
is
driven
that
month.
You
might
see
later
on
when
I
show
this
data,
there
might
be
a
bunch
of
zeros.
Zero
just
means
I
either
didn't
drive
that
vehicle
or
I
just
didn't
drive
it
in
autonomous
mode.
In
either
case,
I
was
not
doing
active
testing
of
the
autonomous
mode
of
the
vehicle.
Now,
as
I
mentioned
earlier,
there
was
a
bit
of
inconsistency.
Prior
to
2018,
when
they
listed
the
disengagement
events,
they
actually
don't
give
the
benefit
of
the
vehicle.
We
don't
know
what
vehicle
was
involved.
We
know
how
many
autonomous
miles
it
drove
that
month,
but
we
have
no
idea
what
vehicle
was
involved.
Starting
in
2018,
that
information
is
now
available.
Now
we
can
match
vehicle
to
the
incident,
which
means
when
we
do
this
analysis,
I'm
going
to
do
it
at
two
different
levels.
One
is
at
an
aggregate
level
where
I'm
going
to
be
looking
at
each
month
all
of
the
vehicles
being
tested
at
that
time.
Then
looking
at
the
incident
rates
overall
in
an
aggregate
measure.
The
second
will
be
then
I
will
zoom
in
at
the
vehicle
level.
I'll
look
at
it
by
VIN.
For
that
data,
I'll
only
be
going
back
to
2018.
For
the
aggregate
level,
I
can
take
all
of
it.
Now,
before
we
get
through
the
analysis,
actually
wanted
to
show
you
some
tools
that
JMP
has
available
that
allowed
us
to
quickly
process
and
accumulate
this
data.
Again,
to
show
you
how
easy
it
is
and
show
off
a
few
features
in
JMP.
Some
of
them
are
really
new,
some
of
them
have
been
around
for
a
little
while.
Let
me
start
by
showing
you
one
thing
that
helped
us,
and
that
was
being
able
to
read
in
data
from
PDFs.
Prior
to
2018,
a
lot
of
these
data
were
compiled
in
PDFs.
Afterwards,
they
put
them
in
an
Excel
file,
which
made
it
a
lot
easier
to
just
copy
and
paste
into
a
JMP
table.
But
for
those
PDFs,
how
did
we
handle
that?
Let
me
give
you
an
example
using
data
from
2017.
This
is
actually
one
of
the
best
formatted
reports
we
see
from
companies.
Some
summaries
here,
some
tables
here
and
there.
This
in
appendix
A
is
the
data
I'm
looking
at.
You
can
see
here,
this
is
the
disengagement
events.
We
have
a
cause,
usually
just
a
category
here.
They
have
the
day,
which
is
actually
the
month.
A
bit
of
a
discrepancy
there,
the
location
and
type.
But
this
is
basically
just
telling
us
how
many
disengagement
events
happen
each
month.
Then
we
have
a
table
here
or
a
series
of
tables
actually
here
at
the
back.
This
is
showing
us
for
each
vehicle,
in
this
case,
it
only
gives
partial
event
information.
There
is
not
a
lot
of
information
available
in
these
early
reports
and
then
showing
you
how
many
autonomous
miles
were
driven
each
month.
How
can
we
put
this
into
JMP?
Well,
I
could
just
copy
and
paste,
but
that's
a
bit
tedious.
We
can
do
better
than
that.
Let
me
come
here.
I'm
going
to
go
to
my
File,
I'm
going
to
go
to
Open.
There's
my
PDF.
I'm
going
to
click
Open
and
JMP
has
a
PDF
import
wizard.
Awesome.
Now
what
it's
going
to
do
is
it's
going
to
go
through
and
look
at
each
page
and identify
whatever
tables
it
finds
there.
It's
going
to
categorize
them
by
the
page
and
what
the
table
is
on
that
page.
Of
course,
when
you
save
it
out,
you
can,
of
course,
change
the
name.
Now,
I
don't
want
every
table
on
every
page.
What
I'm
going
to
do
is
I'm
going
to
go
to
this
red
triangle
on
this
page
and
just
say,
"Ignore
all
the
tables
on
this
page.
I
don't
want
these."
I'll
say,
""Okay,"
I'll
do
the
same
here.
It's
a
nice
summary
table,
but
it's
not
what
I
want.
Then
I
start
saying, "
This
is
the
data
I
want."
Now,
we're
going
to
notice
here,
this
is
formatted
pretty
well.
It's
gotten
the
data
I
want.
If
I
scroll
to
the
next
one,
this
is
technically
a
continuation
of
the
table
from
before.
However,
by
default,
JMP
is
going
to
assume
that
every
table
on
each
page
is
its
own
entity.
What
I
can
do
to
tell
JMP
that
actually
this
is
just
a
continuation
is
to
go
to
the
table
here
on
the
page,
click
the
red
triangle
and
say
for
the
number
of
rows
to
use
as
header,
there
actually
are
none.
This
is
a
way
to
tell
JMP
that
actually
that's
a
continuation
of
the
previous
table.
We'll
check
in
the
data
here,
and
it
looks
like
it
did
it
now.
I'm
going
to
check
here
at
the
bottom
and
I
noticed,
"Oh,
I
missed
that
October
data.
That's
okay.
I'm
going
to
do
a
quick
little
stretch
there
and
boom,
I
got
it."
That's
okay.
You
can
manipulate
the
tables.
I
f
it
didn't
catch
something,
you
can
stretch
and
manipulate
the
table
to
adjust
it.
You
can
also
add
tables
it
didn't
find.
In
this
case,
I
missed
this.
That's
okay.
I'm
going
to
drag
a
box
around it.
Boom.
There's
a
new
table
for
you,
JMP.
I'm
going
to
go
in
here.
It's
going
to
assume
that
there
are
some
header
rows.
Actually,
there
are
none.
Okay,
great.
Now
it's
captured
that
part
of
the
data.
There's
a
bit
of
an
empty
cell
here.
That's
just
a
formatting
error
because
this
is
technically
two
lines,
so
they
didn't
put
this
at
the
top.
It's
okay.
Easy
fix
on
the
back
end.
Now
for
these
tables,
what
we
notice
if
we
go
to
that
is
it
actually
thinks,
"Well,
this
is
actually
one
table."
Unfortunately,
it's
technically
not
correct
because
there
are
two
tables,
but
it's
an
easy
fix.
I
can
simply
go
to
each
one
and
say,
"It's
actually
not
a
continuation
JMP.
This
actually
has
its
own
header."
It
says,
"Okay,"
and
you
can
do
that
for
each
of
these
tables.
I
won't
do
it
for
all
of
them.
I'm
just
doing
this
to
illustrate.
What
we'd
have
to
do
is
we'd
probably
end
up
with
a
bunch
of
tables
here
where
we'll
have
to
horizontally
concatenate.
That's
just
the
way
they
decide
to
format
in
the
report.
But
JMP
has
a
lot
of
tools
to
help
us
with
concatenating
and
putting
tables
together.
But
you
can
see
this
is
a
lot
easier
than
trying
to
copy
and
paste
this
into
JMP,
making
sure
that
the
formatting
is
all
good.
JMP
is
going
to
do
a
lot
of
that
for
us.
Okay,
another
helpful
tool
that
came
out
recently
in
JMP
2017
is,
if
you've
probably
heard
of
it,
the
JMP
workflow.
That
was
super
helpful
because
obviously
we
have
multiple
reports
over
multiple
years.
We'd
like
to
at
least
combine
across
all
the
years
into
two
reports,
one
with
the
disengagement
events,
one
with
the
mileage.
What
we
did
is
we
created
an
initial...
We
followed
some
steps
to
set
up
the
table
in
a
way
that
we
can
then
concatenate
them
together
into
one
table,
and
then
we
saved
it
into
a
workflow.
That's
what
I
have
demonstrated
here.
This
is
a
workflow
builder
that
we
put
together
for
that.
I'm
going
to
demonstrate
it
using
this
data
set.
This
is
particular
for
our
mileage.
What
we
have
here
is
a
table.
This
represents
what
we
would
have
a
raw
output
from
one
of
the
reports.
Here
we,
of
course,
have
it
broken
down
by
VIN
number.
We've
got
a
lot
of
information
here.
We'd
like
to
reformat
this
table.
First
thing
I'm
going
to
do,
I'm
just
going
to
walk
through
each
step.
I'm
not
going
to
show
too
many
details
in
each
step.
You'll
see
what
they
are.
It's
pretty
self
explanatory.
This
first
one
is
going
to
change
the
name
of
this
column
to
vehicle.
That
way
it
matches
a
column
in
our
concatenated
table.
I'm
going
to
go
over,
I'm
going
to
delete
this
total
column.
I
don't
need
that.
Then
I'm
going
to
do
a
stack
across
all
the
dates.
You
can
see
I've
got
that
here.
We
conveniently
called
it
stacked
table,
very
informative.
Now,
one
thing
I
need
to
do
here,
I
put
a
pause
here,
that's
the
little
stop
sign.
In.
That's
because
I
would
usually
need
to
go
in
and
change
the
year.
Now,
something
I
could
do
right
now,
I
couldn't
really
figure
out
a
way
to
get
a
variable,
say
year,
that
you
could
just
fill
out,
put
the
year
there,
and
then
it
automatically
fill
it
in
here.
That's
maybe
something
I
can
go
to
community.jmp.com
,
go
onto
the
wish
list
and
say,
"Hey,
it'd
be
nice
if
I
could
do
this."
But
for
right
now,
I
just
put
in
the
years.
It
was
pretty
easy
to
do
compared
to
doing
this
multiple
times.
Pretty
straightforward.
But
again,
I
can
also
highlight
for
you
how
you
can
actually
go
in
and
adjust
the
script
itself.
You
can
go
in
and
tailor
this
to
your
needs.
What
this
is
going
to
do
is
recode
these
so
it
shows
the
month
and
the
year.
I'll
do
that
real
quick.
There
we
are.
The
next
step
is
going
to
take
this.
Right
now,
this
is
a
category,
it's
a
string,
I
want
a
number.
That's
what
I
do
next.
Now,
this
isn't
pretty.
This
is
just
the
number
of
seconds
since
some
date
in
1900,
I
believe.
Obviously,
that's
not
pretty.
I'd
like
to
show
something
more
informative.
That's
what
I
do
in
the
next
step.
Now,
it
shows
the
month
and
the
year.
I'm
going
to
stop
here.
I'm
not
going
to
continue
because
at
this
point
I'd
have
another
table
open.
This
next
step
would
then
concatenate
the
tables
and
then
close
off
these
intermediate
tables.
What
I'm
going
to
do
is
I'm
going
to
reset
stuff.
I'll
reset,
click
here.
I'm
going
to
reopen
this
table.
I'm
going
to
do
this
just
so
you
can
see
how
fast
this
goes.
Here
I'm
going
to
click
over
here,
I'm
going
to
click
Play,
I'm
going
to
click
Play
again.
Look
how
fast
that
was.
Now,
imagine
doing
this
for
multiple
reports.
How
much
faster
that
is
than
repeating
the
same
steps
over
and
over
and
over
again.
This
workflow
was
really
helpful
in
this
situation.
Now,
I'm
going
to
close
all
these
out
because
it's
time
to
get
into
the
analysis.
Let's
do
that.
I'm
going
to
start
with
the
aggregate
level
data.
Here's
my
table.
I
compiled
across
all
the
data,
all
the
time
periods.
We
have
the
month,
we
have
how
many
disengagement
happened
in
that
month.
I
got
a
column
here
for
the
cumulative
total.
I've
got
here
how
many
autonomous
models
were
driven.
I
got
two
columns
here
that
I'm
going
to
talk
about
in
just
a
second.
You'll
have
to
just
wait.
What
I'm
going
to
do
is
I'm
going
to
go
in,
I'm
going
to
go
to
Analyze.
I'm
going
to
go
under
Reliability
and
Survival,
and
then
I'm
going
to
scroll
all
the
way
down
until
I
reach
Reliability
Growth.
I'll
click
that.
Now
we
have
multiple
tabs
here.
I'm
only
going
to
focus
on
these
first
two
because
these
last
two
concern
if
I
have
multiple
systems.
I'll
revisit
those
when
we
get
to
the
actual
vehicle
information.
For
right
now,
let's
pretend
that
we're
looking
at
the
whole
AV
system,
the
artificial
intelligence
system
in
these
vehicles.
Think
of
it
as
one
big
system.
There
are
two
ways
that
I
can
assess
this.
One,
I
can
do
as
time
to
event,
essentially
how
many
months
until
a
certain
event
happened
or
days
if
we
had
that.
Or
I
could
do
it
via
a
particular
time
stamp.
Basically,
what
was
the
time
at
which
it
occurred?
I
do
have
that
type
of
formatted
data.
I
have
it
at
the
month.
The
month
is
a
fine
timestamp.
It
just
says
in
that
month
I
had
this
many
events
happen.
That's
all
I
need
to
put
in.
I
have
all
the
data
I
need.
I'll
click
OK.
Now,
great
thing
about
this
is
before
you
do
any
analysis,
you
should,
of
course,
look
at
your
data,
visualize
your
data.
It's
nice
because
the
first
thing
it
does
is
it
visualizes
the
data
for
you.
Let's
look
at
it.
One
thing
we're
looking
at,
we're
looking
at
cumulative
events
over
time.
What
we
expect
is
a
behavior
where
early
on
we
might
have
what
I'll
call
a
burn
in
type
period
where
I
have
a
lot
of
events
and
it's
happening.
I'm
tweaking
the
system,
helping
fix
it,
helping
improve
it.
Then
ultimately,
what
I'd
like
to
see
is
this
plateau.
I'd
like
it
to
increase
and
then
flatten
off.
That
tells
me
that
my
number
of
incidents
is
decreasing.
If
it
goes
completely
flat,
that's
great.
I
have
no
more
incidents
whatsoever.
I
wish
the
were
like
that,
it
is
not.
But
we
can
see
patterns
here
in
the
data.
Let's
walk
through.
We
have
a
burn
in
period
here,
early
2015,
and
then
about
mid
2016,
we
flatten
off
until
about
here.
We
see
a
little
blip,
about
summer
of
2016,
something
happens.
We
get
a
few
more
incidents.
We
level
off
again
until
we
get
to
about
here,
about
late
spring
of
2018.
Something
else
happened
because
we
start
going
up
again.
They're
not
very
steep.
This
one's
a
bit
longer.
Then
we
pretty
much
at
here,
we
almost
flattened
out.
We've
reached
the
period
where
we're
really
having
no
incidents
happen,
essentially,
till
the
end
of
2020.
Then
something
happens
in
2021
and
where
we've
reached
essentially
another
burn
in
period,
something's
going
on.
Essentially
what
we've
got
is
four
phases,
if
you
will,
happening
in
the
growth
of
the
system.
Something's
changed
two
or
three
times
to
impact
the
reliability.
Another
way
to
visualize
this,
I'll
run
this
plot.
This
uses
some
data.
I'm
plotting
again
the
cumulative
total.
I'm
also
plotting
something
what
I
call
the
empirical
mean
time
between
failures.
It's
a
very
simple
metric
to
compute.
It's
just
the
inverse
of
the
number
of
disengagements.
It
is
a
very
ad
hoc,
naive
way
to
try
and
estimate
the
mean
time
between
incidents.
But
I
plotted
here
so
that
you
can
see,
you'll
notice
these
four peaks
that
correspond
to
the
bend
in
the
curve.
But
there
are
four
of
them
indicating
these
four
places
where
something
has
changed
in
the
system
to
affect
its
reliability.
What
we
can
do
then
is
try
to
figure
out,
what
are
those
breakpoints?
One
way
you
could
do
that
is
the
reliability
growth
platform
has
a
way
to
fit
a
certain
model.
I'll
pause
here
to
talk
about
the
model
a
bit.
All
of
these
are
actually
the
same
model
with
slight
modifications.
They're
all
what
we
call
a
non-homogeneous poisson process.
That
is
a
fancy
way
to
describe
a
counting
process.
I'm
counting
something,
but
the
rate
at
which
the
accounts
might
occur
per
unit
time
is
changing
over
time.
A
Poisson process
just
means
that
at
a
constant
rate,
so
the
rate
at
which
incidents
would
occur
would
to
be
constant,
that
would
be
equivalent
to
seeing
a
straight
line.
It's
very
easy
to
model,
but
it's
bad
for
reality
because
obviously
we
don't
want
the
rate
to
stay
the
same.
We
would
like
it
to
decrease
to
essentially
zero.
That's
why
we
have
a
non-homogeneous poisson process. We
want
it
to
change
over
time.
Here
we
have
a
model
where
we
can
actually
let
JMP
try
and
figure
out
a
change
point
in
the
process.
If
I
run
it,
what
it's
going
to
do
is
it's
actually
going
to
catch
this
big
piece
and
say,
"Hey,
something
really
changed
there.
For
most
of
it,
it
was
the
same
thing,
but
after
this
point,
it
really
changed."
Now
here
it's
only
going
to
change
one
at
a
time.
I
have
talked
to
the
developer
about,
wouldn't
it
be
nice
if
we
could
identify
multiple
change
points?
A
pparently
that's
a
bit
of
an
open
research
problem,
so
me
and
him
might
be
working
together
to
try
and
figure
that
out.
But
what
I
did
is
I
essentially
eyeballed
it
and
said,
"I
think
there
are
certain
phases.
I
think
there's
about
three
or
four
phases,
and
I
did
it
empirically,
which
is
where
you
get
this
column."
I'm
going
to
run
that
script.
Let
me
show
you
how
I
did
it.
I
come
here
under
redo,
go
to
relaunch,
and
all
I
did
was
I
added
the
phase
column
here.
This
tells
you
that
there
are
different
periods
where
the
reliability
might
have
changed
significantly,
excuse
me,
in
some
way.
If
we think
of
that,
we're
going
to
look
at
the
key
metric
here
as
the
mean
time
between
failure.
We're
going
to
see
early
on,
so
this
is
in
months,
this
here
is
about
three
days,
4-
5,
about
a
week,
and
this
is
about
a
day,
day
and
a
half.
Early
on,
we
have
a
bit
of
a
low
time.
It's
pretty
frequent.
We
can
also
look
here,
I'll
show
you
the
intensity
plot.
That
might
be
another
thing
to
interpret.
What
we're
looking
for
is
we'd
like
the
mean
time
between
failures
to
be
long.
We'd
like
it
to
be
a
long
time
between
incidents,
ideally
infinite.
That
means
nothing
ever
happens,
and
our
intensity
to
decrease.
What
we're
looking
here
is,
we
get
a
bit
of
a
good
start.
About
middle
of
2016,
we're
doing
really
well.
In
fact,
we
get
to
about
a
week
between
an
incident
for
any
vehicle.
There
was
a
bit
of
a
blip,
but
we
primarily
get
back
to
where
we
were
until
we
get
to
the
end
of
2021,
where
now
it's
essentially
about
a
day
between
incidents
for
any
vehicle.
Something
big
happened
here
at
the
end
of
2020
with
these
vehicles
with
this
software
system,
if
you
will.
Again,
you
can
see
here
with
the
intensity,
you
can
almost
do
one
curve
and
we
get
down
to
about
six
or
seven
incidents
per
month.
Whereas
here
it's
almost
30,
essentially,
once
a
day.
We've
been
able
to
look
into
here
and
discover
what's
going
on,
at
least
at
the
aggregate
level.
Before
we
get
to
the
vehicle
level,
I'm
going
to
run
one
more
graph
that's
looking
at,
we've
got
all
these
autonomous
miles.
Could
it
be
that
if
I
drive
it
more
often,
maybe
I
encounter
more
incidents?
Could
that
have
an
effect?
T
here's
a
quick
way
to
assess
that.
Just
using
a
simple
graph.
We'll
just
plot
autonomous
miles
versus
the
total
disengagements.
We
see
here
for
are
a
few
number
of
disengagements,
that
might
be
true.
The
more
you
drive,
the
more
you
might
see.
But
in
general,
long
term,
not
really.
There's
really
no
big,
strong
correlation
between
how
many
autonomous
miles
driven,
how
many
engagements
you
see.
There's
something
else
going
on. T
hat's
actually
what
we
found
in
the
paper
that
I
mentioned
earlier
is
that
the
mileage
impact
was
very
minimal.
Now,
let's
zoom
in
to
the
individual
vehicle.
We're
not
going
to
have
all
the
data,
even
though
I
actually
do
have
it
here.
But
we're
not
going
to
have
complete
data
for
all
of
the
vehicles.
Let
me
break
it
down.
I
have
the
month,
I
have
the
vehicle
identification
number.
Notice
some
of
these,
it's
only
partial.
I
have
here
what
I
call
a
VIN
series.
This
is
very
empirical.
I'm
just
taking
the
first
four
digits
of
the
VIN.
You'll
see
here,
I'm
going
to
scroll
down
a
bit
and
we'll
see.
Let's
see,
maybe
I
will
drag
down
a
little
bit.
There
we
go.
Some
of
these
VINs,
a
lot
of
them
actually
start
with
the
same
four
digits,
2C4R.
I'll
call
them
the
2C4
series.
There's
a
bunch
of
vehicles
that
have
this
as
their
starting
one.
This
identifies
a
particular
fleet
of
vehicles,
at
least
from
an
empirical
view.
If
I
scroll
down,
we're
going
to
run
into
a
different
series,
which
I'm
going
to
call
the
SADH
series.
This
is
the
one
that
was
introduced
about
2021.
That's
when
I
saw
the
Venn
numbers
change
to
the
SADH
designation.
Again,
I
have
how
many
of
miles,
I
have
a
starting
month,
when
did
that
vehicle
start?
I'm
going
to
use
this
to
compute
the
time
to
events.
First,
I'm
going
to
do
a
plot.
I
think
this
is
the
most
informative
plot
you'll
see
for
this
analysis.
What
I've
done
here
is
I've
essentially
created
for
you
a
heat
map.
You
can
see
I've
got
the
heat
map
option
select.
Selected,
I
got
for
each
vehicle
and
over
time,
essentially
a
cell,
and
that's
just
going
to
indicate,
was
I
driven
in
autonomous
mode
anytime
that
month?
I
got
it
color
coded
by
the
series.
These
vertical
lines
correspond
to
the
transitions
between
those
empirical
phases
I
mentioned
earlier.
What
this
is
telling
us
is
basically,
can
we
identify
what
might
have
caused
those
transitions?
Here
we
see
an
initial
series
of
vehicles,
and
it
looks
like
there
wasn't
a
big
change
in
what
vehicles
were
introduced
here.
Maybe
there
was
a
bit
of
a
software
upgrade
for
this
particular
series
that
may
have
introduced
those
new
incidents.
Here
we
see
that
a
new
series
was
introduced,
a
smaller
number
of
vehicles,
maybe pilot
series.
Then
a
bunch
of
them
introduced
about
that
same
period
where
we
saw
the
other
transition.
Here,
this
seems
to
correspond
to
a
new
fleet
of
vehicles
with
maybe
a
slightly
updated
version
of
the
software.
Here,
we
see
a
clear
distinction.
Obviously,
in
2021,
a
completely
new
series
of
vehicles
was
introduced.
We
have
a
bit
of
the
old
vehicles
still
there
in
the
mixture,
but
most
of
them
are
the
new
vehicles.
That
probably
explains
why
we
got
a
new
batch
of
new
incidents.
We
got
a
burn
in
period
for
this
new
series
of
vehicles.
This
is
cool
because
now
we
have
a
bit
more
explanation
as
to
what
was
going
on
with
the
aggregate
data,
which
is
why
it's
important
to
have
this
information.
Now
let's
break
it
down
by
VIN.
I
have
right
here
script
to
indicate
we've
got
a
table
here
and
it's
similar
to
the
table
I
have
previously.
Notice
some
of
these
have
been
excluded
and
this
is
because
if
for
that
particular
vehicle,
the
total
number
of
incidents
was
less
than
three,
the
platform
is
not
going
to
be
able
to
fit
a
model
for
you
because
it
needs
at
least
three
incidents
per
vehicle.
That
makes
sense.
I
have
only
one
or
two,
that's
not
really
enough
information
to
assess
the
reliability.
If
I
have
three
or
more,
now
we're
talking,
I
can
do
something.
I
also
have
the
month
since
the
start.
I
have
some
cumulative
information
there
which
month
it
started.
I'm
going
to
go
ahead
then
and
run
the
platform.
Don't
worry,
I
will
show
you
how
I
did
this.
I'm
going
to
go
to
the
redo,
relaunch.
I'm
going
to
get
rid
of
that.
That's
some
leftover
stuff.
I'm
looking
at
one
of
these
two
last
platforms.
These
are
about
multiple
systems.
Now
we're
thinking
of
each
vehicle
as
its
own
system.
The
concurrent
just
means
I'm
going
to
run
each
vehicle
one
after
the
other.
That's
not
what's
happening
here.
The
vehicles
are
essentially
being
run
in
parallel.
Multiple
vehicles
are
being
driven
at
a
time.
Here
I
need
a
column
to
indicate
the
time
to
event,
in
this
case,
how
many
months
since
the
start
until
this
many
events
happened.
I
have
the
system
ID.
The
one
thing
that's
not
shown
here
is
I
actually
took
the
VIN
series
and
used
that
as
a
by
variable,
which
is
why
we
have
the
where
VIN
equals
so
and
so.
There's
only
going
to
be
two
because
the
one
with
a
little
asterisks
has
no
information
about
incidents.
That's
because
that
was
two
earlier
that
was
prior
to
being
able
to
tie
the
VIN to
the
vehicle.
But
they
were
there
for
completeness,
I'm cancelled
out
of
that.
Now,
what
we're
going
to
see
here
is
a
list
of
models
that
you
could
run.
These
first
four,
I'm
not
going
to
be
able
to
do
because
I
only
have
one
phase.
Essentially,
the
phase
now
corresponds
to
the
VIN
series.
But
there
are
two
that
I
could
run
and
the
only
difference
is,
do
I
want
to
run
a
model
for
all
of
them
saying
these
are
all
part
of
an
identical
system?
Makes
sense.
These
are
all
vehicles,
they
probably
run
the
same
software.
Maybe
I
can
run
a
model
for
all
of
them.
Or
I
can
have
a
model
where
I
fit
it
to
each
individual
vehicle.
Before
I
run
those
models,
let's
take
a
look
at
this
plot.
Again,
start
with
the
visualization
and
what
we
see
plotted
is
all
the
vehicles.
Notice
here
there's
a
bit
of
a
shallow
slope
to
this.
Essentially
there's
a
bit
of
a
steep
curve,
but
then
it
levels
off
pretty
quick.
This
is
a
pretty
good
sign
of
reliability
going
on
here.
I'm
going
to
compare
them,
I'm
going
to
scroll
down
to
the
next
one,
the
set
series.
Now,
initially
the
axes
here,
just
so
you
know,
when
you
run
it
next
time,
the
axes
are
going
to
only
go
to
the
complete
set
of
data.
This
would
actually
be
a
smaller
range.
I
fix
them
so
they
had
the
same
range.
You
can
clearly
see
that
this
is
much
steeper
than
this.
Clearly,
we
have
more
incidents
happening
with
this
new
series
than
this
one.
But
we
can
do
a
quick
model
fit.
I'm
going
to
do
the
identical
system.
Again,
it's
a
non-homogeneous poisson process.
Although
in
this
case,
I'm
going
to
ignore
the
estimates
for
right
now,
if
you
want
to
look
at
that,
you
can.
I'm
going
to
go
straight
to
the
mean
time
between
failure
and
you'll
notice
that
for
all
the
months
it's
pretty
much
flat.
What
it's
essentially
done
is
this
is
just
a
poisson
process.
The
rate
is
constant,
which
is
good
for
modeling,
not
so
good
in
terms
of
assessing
it.
It's
just
saying
across
this
whole
time
for
this
particular
series,
we
pretty
much
reached
for
any
one
vehicle,
a
mean
time
of
about
five
months
between
incidents.
Now,
let's
compare
that
to
what
we
saw
with
the
aggregate
where
it
was
about
a
week,
that's
across
any
vehicle.
It's
just
saying
for
any
vehicle,
it
was
about
a
week
between
an
incident
for
any
vehicle.
Whereas
this
seems
to
be
implying
it's
about
five
months
for
any
one
vehicle.
You
can
think
of
it
as
the
running
in
parallel,
so
you
can
see
it's
staggered.
Any
one
vehicle,
it
could
be
about
five
months.
Again,
this
is
an
average,
there's
a
lot
of
range
between
there.
For
one
vehicle,
it's
a
pretty
long
time.
But
in
aggregate,
they're
probably
staggered
enough
that
it
seems
like
it's
about
a
week
for
any
one
vehicle.
They
can
be
consistent
like
that.
But
this
is
still
pretty
good.
That's
about
five
months
between
an
incident,
a
disengagement
event
here
for
that
series.
If
we
run
to
the
SADH
series
and
do
the
same
model,
let
me
go
here.
There
we
see
clearly
increasing.
I'm
going
to
hide
that.
If
we
look
at
this,
it
says,
"No,"
early
on
we
probably
had
about
two
months.
That's
a
bit
of
a
start
there.
But
we've
dropped
to
less
than
a
month,
almost
two
to
three
weeks
between
incidents.
Clearly,
there's
a
bit
more
work
to
do
on
this
series.
Again,
it
was
just
introduced,
so
this
is
probably
more
of
the
burn
in
phase.
If
we
get
the
2022
data,
we
might
start
to
see
it
level
off
like
it
did
in
the
previous
series.
This
actually
might
flip
and
be
more
of
a
level
curve.
That's
about
all
I
want
to
show
for
these
platforms.
I
can
show
you
some
of
the
individual
distinct
systems,
but
there
are
a
lot
of
systems
here
and
so
it's
going
to
get
crowded
very
quickly.
There's
a
plot
for
each
one.
There's
estimates
for
each
one.
You
can
look
at
the
mean
time
between
failure
for
each
one.
If
there
are
particular
vehicles
you
wanted
to
call
out
and
see
how
they
might
differ,
this
is
what
you
can
do.
You
can
see
some
increase,
some
decrease,
but
overall
more
or
less
flat.
You
can
also
look
at
intensity
plots.
If
you
find
that
more
interpretable
than
the
mean
time
between
failure,
you
have
other
metrics
that
you
can
incorporate
here.
Okay, t
hat's
all
I
want
to
show
for
this
platform.
Now,
of
course,
there's
data
I
didn't
include
here.
For
example,
we
could
break
it
down
by
cause.
For
some
of
this
data,
cause
might
be,
I
just
need
to
take
over
because
it
was
getting
too
close
to
the
side
of
the
the
road.
Or
maybe
the
car
stopped
at
the
stop
sign,
did
what
it
was
supposed
to,
started
rolling,
and
some
other
driver
blew
through
the
stop
sign
coming
the
other
way.
In
which
case,
maybe
that
might
not
necessarily
be
a
reliability
hit.
The
car
did
what
it
was
supposed
to.
Somebody
else
wasn't.
It'd
be
interesting
to
break
it
down
by
that,
also
by
location.
The
number
of
incidents,
you
get
more
when
you're
in
the
city,
maybe
on
the
highway,
something
like
that.
Real
quick,
we
should
look
at
the
mileage
impact.
Again,
same
information,
one
or
two
incidents,
sure,
that
might
change.
But
overall
it's
going
to
be
flat.
The
mileage
impact
on
the
incident
rate
is
minimal.
Of
course,
this
is
just
one
of
many
platforms
available
in
the
reliability
suite.
You
can
see
there's
a
ton
of
options,
very
flexible
for
helping
assess
reliability.
Again,
that's
all
I
have
to
show
you.
Hopefully,
I've
been
able
to
demonstrate
for
you
how
well
JMP
can
help
initiate
discovery
and
analysis.
Hopefully,
you
discovered
a
lot
of
things
about
this
particular
company's
autonomous
vehicles.
I
hope
you
enjoy
the
rest
of
the
conference.
Thank
you.
