I
want
to
thank
the
JMP
steering
committee
and
the
JMP
organizers
for
inviting
Phil
and
myself
to
come
and
present
our
exciting
talk
on
CMC
,
SVEM
,
DOE
,
and
Complexity
:
It
's
All
About
Prediction
.
Want
to
start
by
thanking
Dr
.
Tiffany
Rao
,
she
's
been
involved
with
the
planning
and
numerous
conversations
for
the
work
that
we
're
going
to
present
today
.
Going
to
do
an
overview
,
tell
you
who
Lundbeck
is
,
who
I
work
for
,
and
then
provide
the
background
for
the
DOE
that
we
're
going
to
talk
about
,
which
is
process
development
for
a
biologic
drug
.
Our
case
study
and
what
I
'm
doing
in
traditional
for
what
I
've
started
to
do
for
development
is
start
with
the
first
step
of
doing
DSD
for
mid
-late
stage
development
,
then
follow
that
with
a
second
step
of
doing
augment
with
a
space
-filling
design
.
Then
we
are
hoping
to
prove
to
you
today
that
for
analysis
that
SVEM
allows
us
to
have
better
prediction
for
all
of
this
work
and
allows
us
to
have
better
timelines
for
our
work
that
we
're
doing
.
Lundbeck
is
located
â¦
We
're
headquartered
in
Copenhagen
,
we
're
over
6
,000
employees
in
over
50
countries
,
and
we
are
striving
to
be
the
number
one
in
brain
health
.
The
part
of
the
company
that
I
work
with
is
the
CMC
biologics
and
we
're
basically
located
in
the
Copenhagen
area
and
in
the
Seattle
area
where
I
'm
located
.
Let
's
talk
about
the
background
for
the
DOE
that
we
're
going
to
present
today
.
The
process
that
we
want
to
develop
for
drug
substance
,
for
these
biologics
,
we
start
with
a
cell
of
vials
,
we
take
those
out
of
the
freezer
,
we
then
expand
in
shake
flasks
,
go
bigger
into
culture
bags
,
maybe
a
seed
bioreactor
,
then
to
a
production
bioreactor
.
That
production
bioreactor
goes
approximately
two
weeks
.
We
have
complex
nutrient
feeds
,
we
have
PH
control
,
temperature
control
,
there
's
the
base
that
we
're
adding
.
Once
we
finish
that
14
-day
production
,
we
need
to
figure
out
a
way
to
get
the
cells
that
are
secreting
our
molecule
into
the
supernatant
.
How
do
we
separate
the
cells
from
the
product
?
That
harvest
can
be
a
centrifuge
,
it
can
be
depth
filtration
.
Then
we
pass
it
on
to
our
downstream
colleagues
.
They
first
usually
do
a
capture
step
where
they
're
getting
rid
of
most
of
the
host
cell
proteins
,
the
host
cell
DNA
.
But
then
we
need
to
do
two
polished
steps
where
we
're
then
saying
,
"Okay
,
what
are
the
product
-related
impurities
?
Maybe
there
's
not
the
full
molecule
there
,
so
we
have
to
get
rid
of
those
."
Then
finally
,
we
have
to
make
sure
,
through
ultra
filtration
and
diofiltration
that
we
can
transfer
into
the
buffer
that
it
's
going
to
be
when
it
is
transferred
for
the
patient
's
use
and
it
's
also
at
the
right
concentration
.
You
can
imagine
,
every
step
along
this
way
,
there
are
many
factors
,
there
are
many
knobs
that
we
can
turn
to
control
this
process
,
make
sure
that
it
's
robust
and
we
're
making
the
same
product
every
time
.
When
we
're
focused
on
treating
the
patient
,
we
also
want
to
focus
on
the
business
.
We
can
't
put
all
of
our
development
resources
for
every
molecule
.
We
want
to
right
-size
the
research
that
we
're
doing
at
the
right
stage
of
the
product
.
There
's
many
things
that
could
kill
a
product
,
but
if
we
can
develop
this
in
the
right
time
and
the
right
space
using
these
tools
from
JMP
,
we
can
shift
this
development
timeline
to
the
left
and
we
can
also
reduce
the
amount
of
resources
and
the
cost
to
the
company
.
If
we
're
first
getting
a
molecule
,
that
's
when
you
're
going
to
start
looking
at
your
categorical
factors
.
We
might
be
doing
the
cell
line
screening
.
We
want
to
make
sure
that
we
have
the
right
cell
line
that
's
going
to
last
all
the
way
through
commercialization
.
For
the
downstream
group
,
they
may
be
looking
at
resins
for
both
upstream
and
downstream
,
looking
at
medias
and
buffer
components
and
the
formulations
of
those
.
That
's
when
you
're
making
sure
that
you
have
the
right
thing
,
that
's
going
to
keep
you
going
through
your
development
pathway
.
But
then
once
you
're
in
the
clinic
,
now
you
want
to
really
start
to
gain
understanding
of
the
process
parameters
.
Our
strategy
is
to
start
with
a
development
screening
design
and
we
want
to
be
bold
in
our
level
settings
at
this
stage
and
I
'll
talk
a
little
bit
more
about
that
later
,
for
the
late
stage
development
.
Then
we
can
build
on
what
we
learned
from
the
Definitive
Screening
Designs
by
augmenting
those
designs
with
space
-filling
or
other
designs
so
that
we
really
understand
that
design
space
.
What
's
different
that
we
're
hoping
to
show
now
than
traditional
walks
through
this
pathway
is
that
in
the
past
,
we
've
been
throwing
out
the
factors
that
we
've
said
aren
't
important
.
But
with
modern
designs
and
modern
ways
of
doing
analysis
,
we
can
keep
all
of
the
factors
and
all
of
the
work
that
we
've
done
so
far
and
gain
better
understanding
of
the
whole
process
,
especially
with
biologics
that
are
quite
complex
.
Before
I
pass
the
baton
to
Phil
,
I
just
wanted
to
talk
one
more
about
â¦
Let
's
see
if
I
can
â¦
I
'm
going
to
minimize
this
screen
just
for
a
minute
so
I
can
show
you
this
.
This
is
an
experiment
that
I
did
to
prove
the
power
of
DOE
for
my
boss
.
The
full
data
set
was
an
OFAT
for
PH
,
and
the
response
was
tighter
.
We
wanted
to
do
very
many
different
levels
in
a
wide
range
because
he
wasn
't
sure
at
the
time
that
we
were
going
to
be
able
to
pick
what
the
optimized
level
was
.
But
what
I
wanted
to
show
him
was
that
,
"Okay
,
we
did
this
experiment
,
we
have
all
of
this
data
.
We
were
able
to
model
where
the
optimized
condition
was
,"
and
that
's
shown
in
blue
,
and
that
turned
out
to
be
the
correct
case
.
When
we
tested
the
model
,
that
was
the
optimized
condition
.
Let
's
pretend
now
that
we
're
starting
,
we
don
't
know
that
data
.
If
we
had
picked
a
conservative
range
setting
for
our
experiment
,
our
noise
to
signal
would
be
quite
high
and
so
we
would
have
missed
finding
the
optimized
spot
.
But
if
we
had
picked
a
wider
range
in
our
settings
and
still
with
only
three
points
,
the
model
still
would
have
chosen
the
optimized
spot
.
What
I
'm
going
to
challenge
the
subject
matter
experts
when
you
're
designing
your
DSDs
is
really
be
bold
in
your
range
setting
.
You
will
still
find
the
optimized
spot
and
you
have
to
have
some
knowledge
of
your
process
so
that
you
can
complete
the
design
of
experiment
and
have
all
of
the
runs
at
least
have
enough
signal
that
you
can
measure
and
then
subsequently
model
.
Once
you
learn
from
your
Definitive
Screening
Designs
more
about
your
design
space
,
you
can
come
back
and
then
you
can
be
internal
to
that
space
.
That
's
when
you
augment
with
a
space
-filling
design
.
Now
I
'm
going
to
pass
the
baton
to
Phil
and
he
's
going
to
take
you
through
the
analysis
.
Okay
,
thank
you
.
Thank
you
,
Patty
.
We
're
going
to
talk
about
a
very
nice
and
somewhat
complicated
experiment
that
Patty
and
her
team
run
.
They
do
a
lot
of
great
work
and
they
're
big
advocates
of
DOE
and
JMP
and
I
'm
very
happy
they
let
me
get
to
play
with
them
sometimes
.
It
's
fascinating
work
.
But
before
I
get
into
the
actual
analysis
,
I
wanted
to
talk
about
a
few
relevant
concepts
that
members
of
the
audience
may
or
may
not
be
familiar
with
,
and
that
includes
complexity
.
It
's
a
really
hot
topic
out
there
.
Talk
about
what
is
actually
prediction
.
That
is
a
muddled
concept
to
many
people
.
Then
from
there
,
I
'll
launch
into
talking
about
how
we
analyze
prediction
and
how
we
did
with
Patty
's
experiment
.
Complexity,
a
fellow
named
Daniel
Finelli
from
London
School
of
Economics,
written
much
about
this
and
he
calls
it
"the
elephant
in
the
room
"
that
statistics
and
many
,
what
he
calls
"metasciences
,"
are
ignoring
and
they
're
ignoring
it
at
their
peril
.
I
won
't
get
into
a
lot
of
detail
.
You
can
look
him
up
on
the
internet
,
he
has
a
lot
of
videos
and
papers
.
But
complexity
is
a
huge
problem
.
It
is
staring
science
and
statistics
and
data
science
and
machine
learning
in
the
face
and
it
needs
to
be
dealt
with
.
At
present
,
we
're
not
really
dealing
with
it
directly
in
statistics
.
By
the
way
,
there
are
now
whole
applied
math
programs
based
on
studying
complex
systems
.
My
bottom
line
is
,
complexity
is
real
.
Complexity
requires
new
thinking
.
We
really
have
to
rethink
DOE
and
analysis
.
You
're
going
to
see
that
for
complex
systems,
and
we
also
have
to
understand
something
else
,
systems
theory
101
is
complex
systems
are
defined
by
their
interactive
behavior
.
In
point
of
fact
,
main
effects
are
actually
even
misleading
.
You
have
to
somehow
be
experimenting
in
a
manner
that
you
can
capture
this
interactive
behavior
,
and
you
're
going
to
see
current
strategies
fall
short
of
that
goal
.
Patty
's
already
mentioned
the
CMC
pathway
.
Nowhere
is
this
problem
of
complexity
more
obvious
than
in
bioprocesses
.
You
have
complex
combinations
of
biology
and
chemistry
,
and
interactions
are
everywhere
.
When
I
talk
to
scientists
in
biotechnology
,
they
know
right
up
front
we
're
dealing
with
really
complex
interactive
systems
.
But
first
,
I
need
to
point
out
prediction
.
If
you
're
working
in
CMC
development
work
,
it
's
all
about
prediction
.
The
ICH
guidelines
that
are
used
by
scientists
in
the
CMC
development
work
don
't
specifically
say
prediction
,
but
if
you
read
what
they
say
,
it
's
all
about
prediction
.
Basically
,
you
're
building
processes
to
manufacture
biologics
,
and
with
the
new
cell
and
gene
therapies
,
these
processes
are
becoming
hopelessly
complicated
.
I
personally
rely
heavily
on
the
scientists
to
explain
it
to
me
,
and
they
're
the
people
who
really
make
all
the
decisions
.
I
'm
the
helper
,
and
I
'm
very
happy
to
be
there
as
part
of
it
.
But
it
's
all
about
prediction
.
That
is
not
how
many
scientists
and
even
statisticians
,
have
viewed
CMC
work
.
By
the
way
,
this
applies
to
all
areas
of
science
.
I
'm
focused
with
Patty
on
the
CMC
development
pathway
,
but
prediction
is
important
.
What
is
prediction
?
It
's
muddled
.
It
's
not
clearly
defined
in
disciplines
.
Here
's
what
it
really
is
and
how
I
define
it
.
It
's
a
measure
of
how
well
models
that
you
develop
interpolate
over
a
design
region
.
In
other
words
,
we
're
going
to
fit
a
model
to
what
we
call
a
training
set
,
and
then
we
need
some
way
of
knowing
how
that
model
would
apply
over
the
whole
design
region
.
In
CMC
work
,
especially
late
stage
,
that
is
very
important
.
You
be
able
to
do
that
,
as
many
of
you
know
.
You
really
have
a
training
set
to
fit
the
model
.
That
training
set
in
no
way
can
evaluate
prediction
.
I
know
there
's
a
common
belief
you
can
evaluate
prediction
on
training
sets
.
You
simply
can
not
.
You
must
have
a
test
set
.
Also
I
'll
talk
a
little
bit
about
the
fact
in
dealing
with
scientists
,
and
a
lot
of
it
in
chemistries
and
biologics
.
Again
,
I
do
a
lot
of
it
in
biotechnology
,
but
also
in
other
areas
like
battery
technology
,
material
science
.
It
is
becoming
very
obvious
.
The
kinetics
are
complicated
.
They
're
constantly
changing
over
design
regions
.
The
kinetic
behavior
that
you
see
around
the
boundaries
is
often
very
different
from
what
's
happening
on
the
interior
.
Why
does
this
matter
?
Well
,
the
classic
approach
to
response
surface
,
even
including
optimal
designs
,
relies
upon
what
I
call
boundary
designs
.
Almost
all
of
your
observations
are
around
the
boundaries
of
the
design
region
.
In
point
of
fact
,
whether
people
want
to
hear
it
or
not
,
the
central
composite
design
,
commonly
used
in
response
surface
,
is
about
the
worst
design
you
could
think
of
for
prediction
.
The
interior
of
the
space
is
empty
.
If
you
fit
these
models
on
the
boundary
,
and
then
you
predict
what
's
happening
on
the
interior
,
it
's
not
prediction
,
it
's
speculation
.
You
don
't
know
.
You
have
no
data
.
I
'm
going
to
show
you
in
the
case
study
,
you
're
probably
going
to
reach
some
wrong
conclusions
.
The
boundary
regions
,
indeed
,
often
behave
very
differently
,
and
we
have
a
need
to
reconsider
our
approach
to
designs
.
Another
issue
in
response
surface
and
statistics
is
this
ubiquitous
use
of
full
quadratic
models
.
They
are
not
sufficient
to
model
complex
response
surfaces
.
In
fact
,
they
're
far
from
it
.
Unfortunately
,
I
get
a
lot
of
pushback
from
statisticians
who
claim
it
is
good
enough
.
My
answer
is
,
"Well
,
if
you
actually
use
designs
that
had
sufficient
interior
points
,
you
'd
quickly
discover
they
don
't
fit
well
at
all
.
Again
,
trying
to
measure
prediction
on
the
interior
of
a
design
region
using
boundary
designs
is
futile
.
By
the
way
,
my
good
friend
,
the
late
John
Cornell
and
Doug
Montgomery
,
published
a
paper
on
this
in
1998
,
and
I
'll
be
polite
,
they
were
ignored
.
It
was
actually
somewhat
nastier
than
ignored
by
the
statistics
community
.
They
showed
in
the
paper
that
full
quadratic
models
are
just
not
sufficient
to
cover
a
design
region
.
Patty
mentioned
SVEM
,
self
-validating
ensemble
modeling
.
It
's
an
algorithm
.
I
'm
one
of
the
co
-developers
with
Dr
.
Chris
Gottwald
of
JMP
,
a
person
I
hold
in
very
high
regard
.
I
won
't
get
into
the
algorithm
by
the
way
,
there
are
references
at
the
end
where
you
can
go
and
learn
more
about
it
.
It
has
been
talked
about
at
discovery
conferences
actually
,
going
all
the
way
back
to
Frankfurt
in
2017
.
But
SVEM
is
an
algorithm
that
allows
you
to
apply
machine
learning
methods
.
Machine
learning
methods
are
all
about
predictive
modeling
.
Believe
me
,
people
in
that
field
know
a
lot
more
than
you
may
think
about
prediction
and
apply
them
to
data
from
small
sets
like
DOE
's
.
I
won
't
get
into
SVEM
.
It
's
a
whole
new
way
of
thinking
about
building
predictive
models
,
and
I
think
it
's
in
its
infancy
,
but
it
's
already
proving
very
powerful
and
useful
in
biotechnology
.
Let
's
get
to
the
experiment
.
This
is
actually
a
hybrid
experiment
that
Patty
and
her
team
created
.
There
are
seven
factors
and
there
are
13
responses
.
But
due
to
time
constraints
,
I
'm
only
going
to
focus
on
four
,
and
even
that
's
going
to
be
hard
to
get
it
all
in
.
The
data
and
the
experiment
are
highly
proprietary
.
I
do
thank
Lundbeck
and
Patty
for
actually
allowing
us
to
use
an
anonymized
version
of
this
design
.
I
have
a
lot
of
case
studies
,
some
of
them
similar
to
this
,
and
the
people
who
own
the
data
wouldn
't
even
let
me
discuss
it
if
I
anonymized
it
.
That
was
very
nice
of
them
.
I
think
we
have
a
really
important
story
to
tell
here
.
This
is
a
hybrid
design
.
It
's
comprised
of
a
19
-run
Definitive
Screening
Design
around
the
boundaries
.
Then
it
has
16
space
-filling
designs
on
the
interior
.
There
are
center
points
in
both
parts
of
the
design
.
How
would
we
analyze
this
?
Well
,
what
I
want
to
do
is
discuss
the
strategies
of
analysis
that
are
used
,
the
algorithms
that
are
used
,
and
make
comparisons
to
SVEM
.
I
'll
tell
you
in
advance
,
SVEM
is
going
to
do
very
well
.
Then
we
'll
talk
about
some
of
the
issues
with
the
models
themselves
and
how
we
use
them
.
I
'm
going
to
do
what
most
people
currently
do
.
I
'm
going
to
take
the
boundary
points
,
the
DSDs
,
fit
models
,
and
then
apply
them
to
the
space
-filling
designs
as
a
test
set
and
see
how
well
my
model
interpolates
.
Step
two
,
I
'll
reverse
the
process
.
I
'll
fit
models
to
the
space
-filling
points
,
and
then
I
'll
use
the
DSD
as
a
test
set
and
see
how
well
my
model
actually
extrapolates
a
little
bit
to
the
boundaries
.
Three
is
a
common
strategy
used
in
machine
learning
.
I
'm
going
to
use
a
holdback
test
set
.
I
'm
going
to
take
the
35
runs
and
break
them
up
.
I
did
this
in
a
way
to
make
them
both
equivalent
as
much
as
I
could
into
a
training
set
containing
both
SFD
and
DSD
points
,
and
then
also
a
whole
back
test
set
that
has
a
representation
of
both
.
Then
finally
,
step
four
,
what
many
people
would
automatically
do
.
I
'll
just
fit
models
to
the
whole
data
set
.
In
general
,
I
don
't
recommend
this
because
there
's
no
way
to
test
the
model
.
I
will
say
up
front
,
because
we
do
have
a
lot
of
space
-filling
points
on
the
interior
,
I
'm
more
comfortable
with
this
approach
than
I
am
in
practice
.
But
these
,
I
find
,
are
the
four
basic
strategies
that
would
be
used
.
How
do
I
analyze
it
?
Well
,
if
you
have
a
DSD
,
people
like
to
use
Fit
Definitive
Screening
,
I
'll
look
at
it
,
it
only
applies
to
DSDs
.
Honestly
,
it
's
not
really
a
predictive-
modeling
strategy
,
nor
do
they
claim
it
is
.
But
I
find
people
seem
to
use
it
that
way
.
I
'll
use
Forward
Selection
.
If
you
know
what
the
AICc
statistic
is
,
we
'll
do
that
in
GenReg
,
in
JMP
17
.
Then
we
'll
look
at
something
they
have
in
GenReg
that
's
very
nice
.
That
is
the
SVEM
algorithm
.
I
'm
going
to
use
that
with
Forward
Selection
.
Then
I
'm
going
to
look
at
something
people
may
not
know
.
It
's
a
hidden
gem
in
JMP
.
Something
called
Moving
Average
in
the
Stepwise
platform
.
John
Saul
put
it
there
many
years
ago
.
I
think
he
was
being
very
insightful
.
Then
we
're
going
to
talk
about
SVEM
and
Neural
Networks
.
Basically
,
no
software
does
this
.
I
have
worked
with
the
Predictum
,
some
of
you
know
Wayne
Levin
and
Predictum
to
develop
an
add
-in
to
do
this
.
It
's
currently
the
only
software
available
that
does
this
.
The
SVEM
add
-in
was
used
to
do
the
Neural
Networks
.
I
won
't
get
into
the
add
-in
particularly
,
I
'll
just
quickly
show
people
where
these
things
are
.
Then
finally
I
said
the
fourth
strategy
was
used
to
hold
data
set
because
I
get
asked
about
this
all
the
time
.
I
just
threw
in
some
K
-cross
Fold
validation
to
use
with
the
SVEM
methods
and
some
of
the
other
methods
.
Those
are
the
methods
we
'll
use
and
for
methods
like
Fit
Definitive
,
Forward
Selection
and
Moving
Average
methods
,
we
'll
assume
a
full
quadratic
model
as
that
is
the
tradition
.
The
other
methods
,
again
,
we
're
going
to
use
a
Neural
Network
which
is
more
flexible
.
There
are
four
responses,
and
this
is
really
important
.
I
didn
't
randomly
select
them
.
There
are
four
of
them
and
they
vary
in
complexity
.
Again
,
I
'll
admit
this
is
subjective
.
There
is
no
internationally
approved
measure
of
complexity
and
this
is
based
upon
the
ability
to
model
the
responses
.
Again
,
there
are
13
responses
.
Typically
,
in
CMC
pathway
work
,
there
are
10
-20
,
maybe
more
,
most
of
them
critical
quality
attributes
.
They
are
important
and
they
vary
within
the
experiment
from
some
are
fairly
low
in
complexity
,
some
are
very
high
,
very
difficult
to
model
.
Frankly
,
in
those
cases
,
Neural
Networks
are
basically
your
only
option
.
So
pay
attention
to
this
because
this
complexity
turns
out
to
be
very
important
in
how
you
would
go
about
modeling
.
Then
the
question
is
if
I
'm
going
to
evaluate
prediction
,
well
,
how
do
I
do
that
?
Remember
,
I
prefer
prediction
be
on
an
independent
test
set
with
new
settings
of
the
factors
.
That
's
how
we
judge
interpolation
.
Well
,
something
called
the
Root
Average
Square
Error
or
RASE
scores
is
very
common
.
This
is
the
standard
deviation
of
prediction
error
.
Again
,
it
's
commonly
used
to
judge
how
well
you
predict
.
Smaller
is
better
,
obviously
,
but
there
is
a
problem
with
it
that
we
've
particularly
uncovered
,
especially
in
simulations
.
Models
with
low
RASE
scores
often
have
substantial
prediction
bias
in
them
.
In
prediction
,
there
really
is
still
a
bias
-variance
trade
-off
.
So
how
do
we
evaluate
bias
?
Well
,
there
's
no
agreed
upon
approach
to
that
either
.
But
the
easiest
way
and
the
most
visual
way
is
actual
by
predicted
plots
on
a
test
set
.
Ideally
,
if
you
were
to
fit
a
slope
to
the
actual
bi
-predicted
plot
,
I
'll
show
an
example
.
The
ideal
prediction
equation
that
a
slope
would
be
one
with
an
intercept
of
zero
.
The
farther
the
slope
is
from
one
,
the
greater
the
bias
.
For
purposes
of
demonstration
,
I
'm
going
to
set
a
specification
of
0
.85
-1
.15
with
a
target
of
1
for
the
slope
.
If
you
can
stay
within
that
range
,
then
I
'd
say
you
probably
have
acceptable
amounts
of
bias
.
In
reality
that
happens
to
be
more
of
a
subject
matter
issue
.
Then
finally
I
said
,
"Well
,
you
can
fit
a
slope
to
the
actual
bi
-predicted
plot
.
There
's
an
additional
problem
."
The
predictor
is
the
predicted
values
.
They
have
a
lot
of
error
in
them
.
So
this
is
actually
an
errors
and
variables
problem
,
which
is
not
commonly
recognized
.
But
JMP
17
has
a
really
nice
solution
.
It
's
called
the
Passing
-Bablok
modeling
algorithm
and
it
's
been
well
-established
,
especially
in the
biopharma
.
This
fits
a
slope
,
taking
into
account
errors
in
X
,
the
predictor
.
So
how
does
it
work
?
Well
,
it
fits
a
slope
.
If
you
look
on
the
left
,
you
'll
see
the
slope
is
about
0
.5
.
We
have
strong
bias
.
There
's
a
lot
of
prediction
bias
.
What
I
really
like
in
the
application
in
JMP
,
they
give
you
the
reference
line
.
The
dashed
blue
line
is
the
ideal
line
slope
of
one
,
intercept
of
zero
.
On
the
left
,
our
predictive
model
is
showing
a
lot
of
bias
.
It
's
systematically
not
predicting
the
response
.
To
the
right
,
is
a
case
where
there
's
actually
a
small
amount
of
bias
in
general
,
that
would
be
acceptable
.
By
the
way
,
these
were
picked
as
one
's
models
that
had
relatively
low
overall
RASE
scores
.
These
are
called
the
Passing
-Bablok
slopes
and
they
are
integral
to
how
I
evaluate
prediction
,
the
overall
RASE
and
the
slopes
.
What
I
'm
going
to
do
at
this
point
,
I
'm
going
to
actually
go
over
to
JMP
,
if
you
don
't
mind
.
I
'll
make
a
quick
change
in
the
screen
here
and
I
'll
make
this
as
big
as
I
can
for
everybody
.
Overall
in
this
exercise
,
I
fit
close
to
140
models
and
I
did
them
all
individually
and
evaluated
them
.
Yes
,
it
took
quite
a
while
and
I
'm
going
to
show
a
graphic
to
try
to
summarize
the
results
for
the
different
methods
.
I
'm
going
to
open
a
Graph
Builder
script
.
I
'll
make
this
as
big
as
I
possibly
can
for
everyone
.
I
'm
using
some
local
data
filters
,
to
define
the
display
.
Notice
we
have
four
training
scenarios
.
I
'll
start
with
where
the
DSD
is
the
training
set
.
We
fit
models
to
the
boundary
and
then
we
evaluate
them
and
how
they
predicted
the
space
-filling
design
points
.
Y2
is
the
easy
response
.
I
expected
all
approaches
to
do
well
,
they
did
.
Notice
I
set
these
spec
limits
and
that
's
0
.85
-1
.15
all
fell
within
that
allowable
region
.
Two
of
the
methods
that
did
well
,
I
particularly
liked
the
moving
average
,
so
it
did
pretty
well
.
None
of
them
had
a
slope
of
exactly
one
.
The
DSD
points
don
't
exactly
predict
what
's
going
on
in
the
space
-filling
design
points
,
but
they
all
did
relatively
well
.
Now
we
'll
go
to
moderate
complexity
.
Now
you
start
to
see
some
separation
.
It
's
getting
harder
to
model
the
surface
.
Again
,
I
'm
using
this
interval
of
0
.85
-1
.1
.
I
'm
looking
on
the
y
-axis
at
the
RASE
score
standard
deviation
of
prediction
.
On
the
x
-axis
,
I
'm
looking
at
slope
.
For
Y1
,
using
the
DSDs
to
predict
the
space
-filling
design
points
as
the
test
set
.
The
only
models
that
really
performed
well
were
the
Neural
Networks
with
SVEM
.
By
the
way
,
the
code
is
NN
is
Neural
Network
,
H
is
number
of
hidden
nodes
.
We
have
models
with
varying
levels
of
hidden
nodes
and
I
simply
evaluated
RASE
scores
and
slope
.
We
go
to
more
complexity
.
Now
Y3
has
high
complexity
.
It
is
hard
to
model
.
The
lowest
RASE
scores
were
the
methods
you
see
on
the
lower
right
,
but
you
can
see
there
's
substantial
prediction
bias
.
I
felt
overall
the
best
combination
of
low
-bias
and
RASE
score
were
Neural
Networks
,
particularly
one
with
27
hidden
nodes
.
Then
finally
number
four
is
high
complexity
.
We
fit
the
model
to
the
DSDs
and
applied
it
to
the
space
-filling
points
.
I
didn
't
think
any
of
the
models
did
great
.
All
of
them
showed
some
prediction
bias
.
Maybe
the
best
performance
was
a
Neural
Network
with
12
hidden
nodes
.
It
had
the
lowest
RASE
score
,
but
still
,
there
were
some
issues
with
bias
.
So
that
's
one
strategy
.
Well
,
what
if
I
were
to
do
the
opposite
?
I
fit
the
model
to
the
space
-filling
points
and
then
apply
them
to
the
boundary
DSD
points
.
Again
,
let
's
start
with
the
easiest
case
.
Y2
really
does
.
It
's
a
pretty
simple
response
.
Actually
,
the
SVEM
method
in
GenReg
using
SVEM
and
Forward
did
very
well
.
The
next
best
I
thought
was
a
Neural
Network
with
10
.
Remember,
there
's
a
little
bit
of
extrapolation
going
on
here
.
Finally
,
Y1
with
moderate
complexity
.
Again
,
only
the
Neural
Networks
did
well
.
As
we
go
up
in
complexity
,
increasingly
just
the
Neural
Networks
are
working.
You
'll
find
similar
results
for
the
other
approaches
.
I
won
't
show
all
of
them
,
they
're
covered
in
the
notes
.
But
the
general
conclusion
by
the
way
,
is
that
when
you
use
the
boundary
points
as
a
test
set
or
you
use
the
space
-filling
designs
as
a
test
set
and
try
to
predict
the
other
,
they
're
just
not
doing
as
well
as
they
should
.
In
other
words
,
as
I
said
earlier
,
the
boundary
points
,
the
DSD
points
and
the
space
-filling
design
points
,
there
are
differences
in
their
kinetic
behavior
that
we
're
not
picking
up
.
The
only
way
we
're
going
to
pick
it
up
is
to
actually
fit
models
over
the
whole
design
space
.
We
did
do
that
by
the
way
.
I
should
just
quickly
show
you
.
I
used
the
whole
data
and
we
fit
models
and
we
actually
did
pretty
well
.
I
didn
't
show
the
Passing
-Bablok
slopes
.
I
will
just
quickly
do
a
little
more
work
with
JMP
for
those
who
are
interested
.
The
Passing
-Bablok
slopes
can
be
done
in
Fit
Y
by
X
.
I
will
admit
we
wrote
a
script
and
added
it
to
the
predictive
add
-in
to
do
this
in
Fit
Y
by
X
,
but
you
can
easily
do
it
yourself
.
Here
,
and
I
'll
pick
one
of
the
cases
,
is
the
DSD
data
and
I
'll
pick
Y1
.
How
did
we
do
fitting
models
?
If
you
look
in
the
menu
,
there
's
the
Passing
-Bablok
.
I
strongly
suggest
you
look
at
it
.
A
lot
of
regression
problems
are
errors
in
variables
.
How
did
the
method
do
it
overall
?
I
want
to
explain
something
else
.
The
orange
points
are
the
DSDs
,
the
boundaries
.
The
blue
points
are
the
space
-filling
design
points
.
Here
I
fit
models
to
the
DSD
and
the
Passing
-Bablok
slopes
are
being
fit
to
the
space
-filling
design
points
.
Overall
,
the
best
performance
was
turned
in
by
the
DSDs
.
There
's
one
of
them
here
.
It
's
Saywood
6
.
Another
one
that
had
â¦
I
forgot
what
it
was
.
Let
me
widen
this
out
for
you
.
Nineteen
.
Notice
the
slope
is
close
to
one
,
but
you
can
clearly
see
there
is
some
bias
.
In
other
words
,
you
can
see
an
offset
between
the
fitted
slope
and
the
ideal
slope
,
the
dashed
blue
line
.
This
is
pretty
typical
overall
.
I
'll
just
very
quickly
show
you
.
If
you
have
JMP
Pro
and
you
want
to
do
SVEM
using
linear
models
,
just
go
to
Fit
Model
,
Recall
.
This
is
a
full
quadratic
model
.
You
could
do
others
.
Go
to
GenReg
and
then
under
estimation
methods
.
There
's
SVEM
Forward
.
There
's
SVEM
Lasso
.
These
work
very
well
.
From
a
lot
of
work
in
these
methods
,
I
still
find
SVEM
Forward
gives
you
the
best
results
.
The
Lasso
tends
to
give
you
a
lot
of
biased
results
on
test
sets
in
particular
.
If
you
're
interested
in
model
averaging
,
if
you
have
JMP
standard
,
just
going
to
hit
recall
again
,
just
go
to
the
Stepwise
platform
.
Didn
't
do
it
.
Stepwise
.
I
won
't
run
it
.
It
will
take
too
long
because
model
averaging
uses
best
subsets
regression
.
It
's
time
-consuming
,
but
it
's
there
.
Again
,
Neural
Networks
with
SVEM
,
you
have
to
have
the
Predictum
add-
in
to
do
that
.
There
's
a
link
to
it
if
you
're
interested
.
At
this
point
,
I
'm
going
to
not
do
too
much
more
analysis
.
Again
,
you
can
go
through
and
look
at
the
various
slopes
for
the
various
responses
and
you
can
see
many
of
these
methods
resulted
in
highly
biased
slopes
.
In
other
words
,
the
DSD
points
and
the
space
-filling
designs
are
too
different
.
We
've
really
got
to
understand
we
need
to
fit
models
over
the
entire
design
region
.
At
this
point
,
I
'm
going
to
just
finish
up
.
By
the
way
,
there
is
enough
material
here,
and
I
do
have
basically
many
talks
that
are
combined
into
here
.
I
apologize
,
but
I
think
there
's
an
important
message
here
.
By
the
way
,
I
'm
just
showing
slides
with
the
Passing
-Bablok
slopes
.
Then
finally
,
I
want
to
just
give
you
some
final
thoughts
.
I
think
we
really
need
some
new
thinking
in
statistics
.
We
don
't
have
to
throw
out
everything
we
've
been
doing
.
I
'm
not
saying
that
.
The
most
important
is
we
are
in
the
era
of
digital
science
.
Digital
chemistry
,
digital
biology
,
digital
biotechnology
are
here
.
They
're
not
tomorrow
.
We
've
got
far
more
automation
.
Lots
of
great
,
especially
in
biotechnology
,
pilot
and
bench
scale
devices
that
scale
nicely
,
where
we
can
do
lots
of
experiments
.
The
problem
is
complexity
.
We
need
to
think
differently
.
Machine
learning
methods
via
SVEM
are
very
important
for
fitting
these
complex
systems
.
We
need
to
get
away
from
the
response
surface
approaches
that
really
haven
't
changed
.
Maybe
we
've
got
computers
and
some
new
designs
.
I
think
DSDs
are
really
very
clever
.
We
have
optimal
designs
,
but
they
suffer
from
the
fact
they
're
boundary
designs
and
people
keep
insisting
on
full
quadratic
models
.
That
's
a
mistake
,
as
I
've
tried
to
show
briefly
in
the
talk
,
and
you
will
be
able
to
download
the
talk
,
you
can
see
how
poorly
these
methods
generally
did
with
the
complex
responses
.
As
far
as
I
'm
concerned
,
we
need
new
types
of
optimal
designs
.
At
a
minimum
,
these
need
to
accommodate
a
lot
of
factors
.
Patty
,
by
the
way
,
without
getting
into
details
,
has
run
a
DSD
â¦
Not
a
DSD
.
You
did
space
-filling
design
with
18
runs
.
Given
they
have
Amber
Technology
available
,
if
you
know
what
that
is
,
they
can
do
it
.
Why
do
we
need
that
?
Because
these
systems
are
interactive
.
We
need
to
stop
thinking
they
're
a
minor
part
of
the
equation
.
Main
effects
do
not
describe
the
behavior
of
a
complex
system
.
Its
interactivity
is
what
drives
the
behavior
.
We
need
to
cover
the
interior
of
the
design
region
.
Yes
,
we
would
like
to
cover
the
boundaries
too
.
We
don
't
want
to
be
specifying
a
model
.
Optimal
designs
require
you
specify
what
is
usually
a
full
quadratic
model
.
We
need
to
get
away
from
that
.
Space
-filling
designs
,
by
the
way
,
are
optimal
designs
that
do
not
require
a
model
be
specified
.
But
they
're
not
the
total
answer
.
We
need
to
cover
the
design
space
.
We
need
to
give
the
user
a
lot
of
input
that
would
be
scientists
on
how
they
distribute
the
points
.
The
work
of
Lu
Lu
and
Anderson
-Cook
point
the
way
.
I
won
't
have
time
to
get
into
that
.
That
's
another
topic
.
We
need
to
be
able
to
easily
combine
our
design
with
other
data
.
That
includes
engineering
runs
,
GMP
runs
,
even
models
from
partial
differential
equations
and
simulations
.
Especially
if
you
want
to
get
into
digital
twins
,
you
've
got
to
be
able
to
do
that
using
what
I
call
meta
models
.
Then
finally
,
Patty
mentioned
this
,
so
I
wanted
to
bring
it
up
.
The
standard
practice
in
design
of
experiments
,
assuming
that
somehow
you
've
got
to
screen
out
factors
is
actually
a
really
high
-risk
,
no
-reward
strategy
in
complex
systems
.
You
will
regret
it
.
You
will
someday
,
at
a
later
stage
,
come
back
and
have
to
redo
experimental
work
.
I
've
seen
this
time
and
again
.
In
complex
systems
,
this
idea
that
there
are
active
and
inactive
factors
is
simply
wrong
.
They
all
matter
at
some
level
somewhere
in
the
design
space
.
Frankly
,
with
our
modern
tools
,
you
don
't
need
to
do
it
anyway
.
Also
,
something
else
people
do
reflexively
reduce
linear
models
.
We
've
shown
in
our
research
in
SVEM
.
Also
,
a
nice
paper
by
Smucker
,
Edwards
,
and
we
showed
reducing
models
degrades
prediction
.
Why
?
Because
you
're
making
your
model
stiffer
and
stiffer
,
it
's
not
going
to
interpolate
well
.
I
will
stop
at
this
point
and
there
are
some
references
at
the
end
.
