All
right.
Hello
everyone.
My
name
is
Scott
Allen.
I'm
a
Systems
Engineer
at
JMP.
I'm
going
to
be
presenting
a
neural
network
tuning
add-
in
that
I've
developed
really to
help
develop
neural
network
models
in
an
automated
way
to
build
many
models,
find
out
which
parameters
are
most
effective
in
building
the
best
model.
What
I'd
like
to
do
is go through
just a little bit of background
on
motivation
for
building
this
add-
in.
I'm
going
to
do
maybe
a
walkthrough,
the
description,
what's
going
on
in
the
background
when
you
run
the
add-
in
and
then
show
a
use
case
for
it.
A
sample
workflow.
Then just really wrap
up
with
some
lessons
learned.
This
was
my
first
big
scripting
project,
so
certainly
learned
a
lot
along
the
way.
I'd like
to
share
that
and
then
as
well
as
what's
next.
First,
just
I
want
to
take
3, 4 minutes here
just
on
the
background
and
motivation.
A lot
of
other
machine
learning
platforms in JMP
have
some
sort of
parameter
tuning
function,
whether
it's
a
tuning
table
that
you
create
outside
the
platform,
like
in
bootstrap
forest
or
boosted
trees,
or
an
integrated
tuning
design
like in
support
vector
machines
and
the
new
XGBoost
platform.
But
there's
not
one
for
neural
networks.
That
led
to
some
customers
that I work with asking about it.
"Is
this
something
that's
available?
Are
there
scripts
out
there
that
we
can
use?
Is
there
an
add-
in?"
There
are
a
few
things
out
there,
but
maybe
not
everything
in
one
place.
Then
on
a
personal
level,
I've
never
undertaken
a
large
scripting
project.
I've
done
some
small
JSL
scripts
to
automate
workflows,
or
to
clean
up
data,
but
I
had
never
built
an
application.
This was, I
really
wanted
a
project
to
help me learn.
I
used
this
as
an
excuse
to
learn
JSL.
The
neural
network
platform
in JMP
is really
powerful,
and
so
I'll
just
launch
it
really
quick.
I'm
sure
if
you've
seen
it...
I
will
mention
I
am
using JMP PRO
version
17.1
.
That's
where
all
the
testing
has
happened
for
this
add-
in.
I
have
not
gone
backwards
many
versions.
I
would
suggest
if
you
are
going
to
use this
you
need
to
have
JMP
PRO
and
at
least
version 17.1.
The
JMP
neural
network platform
is
really
powerful
and
I
just
want
to
take
a
few
minutes
to
go
through
what
this
looks
like.
I
think
everybody
has...
If
you've
used
neural
networks,
you've
gone
through
this workflow
and
it's
a
really
nice
flexible
platform
for
developing
these
models.
You
can
start
off
with
a baseline model
and
maybe
you're
not
satisfied
with
the
performance
of
your
model
in
this
case.
You
can
just
go
back
to
the
model launch
and
you
can
adjust
the
parameters
here.
You
can
go
and
generate
another
model.
It's
easy to
start
changing
these
parameters
and
finding
models
that
might
be
the
most
appropriate
for
your
system.
But
there
are
a
lot
of
options
in
here.
You've
got
different
number
of
nodes,
you've
got
one
or
two
layers,
there's
boosting
options
as
well
as
a
lot
of
different
fitting
options.
You
don't
always
know...
It's
not
always
clear
which
of
these
parameters you need to change
in
order
to
get
the
best
model
for
the
data
table
that
you're
working
in
and
the
predictors
that
you're
using.
It's
not
always
clear
if
adjusting
a
parameter
is
going
to
lead
to
a
better
model.
You
just
adjust
up a
nd
down,
maybe
take
big
steps
to
figure
out
what's
going
on.
It does
require
a
lot
of
manual
adjustments
by
the
user
in
order
to
generate
these
models.
They
are
pretty
fast,
so
it's
not
time-
consuming
to
create
these
but
it d
oes
require
a
lot
of
clicks.
My
goal
here
and
some
of
my
inspiration
was
really
to
provide
a
single
platform
for
tuning
and
evaluating
those
neural
network
models.
I
was
also
inspired
there
were
some
tuning
scripts
out
there.
Nick
Shelton
had
a
script
that
would
help
you
build that table
and
run
it
and
evaluate
those
models.
Mark
Bailey
had
developed
an
add-
in
for
this
as
well,
and
there
were s
ome
community
posts
that
were
trying
to f
igure
out
how
to
do this.
Those
were
all
some
starting
points
for
this
add-
in.
Then
I
really
like
the
XGBoost
add-
in
interface.
It's
really
clean,
it's
got
some
graphical
outputs
and
it's
really
easy
to
navigate
and
lets
you
auto
tune
those
networks.
The
add-
in
that
I'm
developing
isn't
auto
tuning,
it's
really
more
of
a
brute
force
method.
Just
build
lots
of
models
and
then
have
a
graphical
interface
there
to
help y
ou
find
the
best
one
for
your
system.
Let's
go
on
and
move
on t
o
just
what
the
add-
in
does
and
how
to
use
it.
To
launch
the
add-
in,
just
like
most
add-
ins,
you
go
to
the
add-
in
menu,
and
it's
called
Neural
Network
Tuning.
That
takes
you
to
the
launch
dialog.
This
gives
you
a
column role
interface
that's very
similar
to
many
other
platforms.
We
can
just
select
all o
f
our
factors
here
and
our
response.
Currently,
it
allows
a
single
Y
response,
and
it
can
be
either
continuous
or
categorical.
The
add-
in
currently
only
has
the
two
validation
methods,
so
you
can
specify
a
validation
column
with
both
validation
and testing.
Settings
if
you
have
that,
if
you
don't,
it's
just
going
to
do
a
random
holdback,
the
default
for t
he
neural
network.
Then
you
can
specify
and
toggle
or
toggle
the
informative
missing
as
well.
But
we'll
just
go
with
this
for
now.
We
click
OK,
and
that
takes
you
to
the
tuning
launch.
This
tuning
dialog,
first,
gives
you
a
little
bit
of
information
about
what's
in
the
data
table
and
what
you
specified
as
column
roles.
Right
now
we've
got
the
data
table here,
the
model
validation
method,
as
well
as
whether
or
not
we
have
informative
missing
on
or
off.
Then
we've
got
the
DOE
option.
What
this
add-
in
will
do,
is
look
at
all
the
parameter
ranges
that
you've
set
and
create
a
fast,
flexible,
space-
filling
design.
Really
the
only
design
specification
you
need
is
to
set
the
number
of
trials
in
this
design.
In
this
case
we've
got
by default
it's
going
to
be
set
to
20,
and
there are
some
rules
of
thumb.
Maybe
you
want
to
specify
at
least
10
treatments
in
this
design
for
every
factor
that
you're
looking
at,
but
it's
completely
up
to
the
user,
and
so
we'll
come
back
and
adjust
that
if
we
need
to.
You
can
also
replicate
this
design
with
a
different
random
seed.
In
some
cases
if
you
want
to
see
how
robust
you
might
be
to
certain
parameters,
you
can
run
the
same
neural
network
model
20
times,
but
all
with
different
random
seeds
to
see
how
robust
the
model
is
to
that.
Then
you
can
toggle
whether
or
not
you
want
to
see
the
DOE
dialog
that
shows
up.
Once
you've
gone
through
the DOE
options
here,
you
can
come
down
to
the
neural
network
tuning.
I
tried
to
stay
pretty
true
to
the
original
platform
in
the
neural
network
by
using
similar
language
and
options
here.
This
is
where
you
go,
a
nd
instead
of
specifying
a
single
value
for
one
of
these
parameters,
we're
going
to
specify
a
range.
Also,
I'm
limiting
a
little
bit
the
available
options
for
the
type
of
neural
network
that
you're
going
to
create.
In
this
case
you
can
generate
a
single
hidden
layer,
you
can
generate
two
hidden
layers,
or
one
hidden
layer
with
boosting.
Currently,
the
neural
network
platform
doesn't
allow
a
second
layer
with
boosting.
If
you
try
to
do
that
in
the
neural
network
platform,
it
will
give
you
a
warning
saying
that
it's
going
to
ignore
that
second
layer.
In
this
add-
in,
I
just
make
it
so
that
you
zero
out
all
those
second
layers
when
you
activate
boosting.
In
this
case
we're
going
to e
xplore
this
data
table
a
bit
and maybe
we're
going
to
set
some
wide
range
of
tuning
parameters.
I'm
going
to
go
from
0-9
on
each
of
those
activation function
number
of
nodes.
Over
on
the
fitting
options,
you
don't
have
to
necessarily
know what
might
be
best
for
your
system,
you
can
just
select
them
to
be
part
f
the
DOE,
or
in
the
case
of these
transform
covariates
and
robust
fit
options.
You
can
toggle
them
to
always
be
off,
always
be
on,
or
include in the DOE.
We'll
include
both
of
those
in
the
DOE
for
this
example.
Penalty
methods.
You
can
test
any
or
all
the
different
methods.
If
you
don't
specify
any
penalty
method,
it
will
run
the
default
penalty
method
squared.
But
we're
going
to
check
all
of
those.
You
can
also
specify
the
number
of
tours
per
model.
In
retrospect,
I
probably
didn't
need
to
add
this
but because
you
could
do
something
similar
by
just
replicating
all
of
the
designs
and
seeing
the
results
from
each
of
those
models.
But
it's
here
in
case
you
want
to
adjust
the
number
of
tours.
You
can
also
specify
a
random
seed.
In
this
case,
I've
got, 3,4,5,6.
Six different
factors
in
this
design,
and so
I'm
just
going
to
maybe
turn t
his
up
to
60.
I'm
going
to
create
60
models.
Before
I
click
Run,
I'm g
oing
to
talk
a
little
bit
about
what's
going
on
when
you
click
Run.
The
first
thing
that
happens
when y
ou
click
Run,
is
it's
going
to
create
a
factor
table
with
only
the
selected
parameters.
We select it.
In
this
case
it'll
create
a
factor
table
with
single-
layer
nodes,
all
of
the
different
fitting
options.
It's
not
going
to
put
the
second l
ayer
or
any
of
the
boosting
parameters
into
the
factor
table.
Then I'm
going
to
create
a
response table
where
the
training,
validation,
and
testing
R-
squared
values are specified
a
nd
then
it's
going
to
combine
those
into
a
single
table.
This
is
what
the
sample DOE
table
will
look
like.
It's
just
going
to
have
the
responses
as
the
R-
squared
values
and
then
all
the
different
parameters.
Then
these
just
get
passed
into
the
neural
network
platforms
row-
by-
row,
and
the
results
show
up
in
the
data
table.
Just
like
we're
saying,
each
of
those
runs
in
the
design
is
going
to
go
into
the
neural
network
model
platform
sequentially
and
then
we
have
a
little
dialog
box
that's
going
to
indicate
the
overall
process.
In
the
case
of
really
long-
running
neural
networks,
it's
good
to
just
see
how
quickly
the
whole
design
is
going
to
go.
I
would
recommend
before
specifying
a
large
number
of
trials
in
this
design
that
you
run
one
or
two
ahead
of
time
just
to
see
how
long
they're
going
to
take.
If
you
have
fairly
small
data
set,
then
they
go
pretty
quickly,
but
if
you
have
a
complex
data
set,
they
can
take
sometimes
quite
a
while
to
run.
In
this
case
I'm
just
using
a
sample
data
set
from
the
sample
data
set
library.
This
tablet
production.
It's
not
the
best
data
set
to
run
for
neural
networks
just
because
there
aren't
a
whole
lot
of
observations,
but
I'm
using
it
just b
ecause
the
neural
networks
run
really
quick.
In
this
case,
if
we're
satisfied
with
all
of
the
specifications
that
we
have
in
our
model,
we
can
just
go
ahead
and
click
Run.
What
this
has
done
is
it's
created
that
design
table
and
now
it's
passing
through
each
of
those
trials
into t
he
neural
network
platform
and
giving
an
output.
If
we
look
at
the
home
window,
all
this
has
done
is
it's
taken
that
tuning
result
table
and
I've
hidden
it
and
I
just
bring i
t
out
into
the
same
window
here.
Let's
just
go
on
to
the
next
one.
We
can
work
through
that
data
table
here,
so
it's
interactive.
What
you
first
see
up
here
is
graph
builder
report.
T
his
shows
you
all
of
your
different
parameters,
so
the
different
activation
functions
as
well
as
the
number
of
nodes
is
the
scale,
and
how
the
validation
R-
squared
changes
with
respect
to
each
of
those
individual
parameters.
In
the
case
of
categorical
parameters,
you g
et
a
box
plot
showing
the
difference
in
the
values
this
way.
We
can
see
in
this
case
maybe
for
robust
fit,
it's
got
a
wider
range,
but
on
average
it's g
oing
to
be
better.
The
median
value
is
better.
Is
higher
R-
squared.
We
can
also
look
down
here
at
the
data table,
it's
going
to
sort
the
data
table
by
the
validation
R-
squared
and
it
just
shades
those
a
deeper
purple
depending
on
the
value.
Let's
see.
We can use...
One
reason
I
just
kept
this
data
table
as
it
is,
is
we
can
explore
it
just
like
we
explore
any
data
table
in
JMP,
so
you
can
look
at
column
headers
or
you
can
launch
graph
builder
or
do
various
analyses
on
this
data
table
as
it
is
right
here.
There
are a
few
additional
data
columns
or
columns
that
are
added
to
this
data
table.
One,
it
records
the
random
seed.
If
you
want
to
reproduce
this,
you
know
what
random
seed
was used
in
generating
the
n
eural
network
models,
shows
you
the
informative
missing,
shows
you
the
elapsed
time
for
each
of
the
models.
If
you
are
building
large
models
or
you
might
want
to
minimize
the r
untime,
then
you
can
get
an
estimate o
f
how
long
it
might
take
for
you
to
run
additional
models.
nDOE
just
records
what
DOE
was
run.
This
was
the
first,
so
it
gets
a
1.
As
you'll
see
we
can
run
a
second
and
third,
and
you
can
then
use
those
as
some
graphing tools.
Also
tells
you
the
type
of
neural
network
that
was
run.
Whether
it
was
single-
layered,
double-
layer,
or
a
single-
layer
with
boosting.
There's
a
couple
of
other
columns
that
are
added.
This
factor
column
just
lets
you
keep track
of
the
factors
that
were
used
in
the
design
in
case
you
want
to
compare
models
that
have
different
factors
that
were
used
instead
of
just
parameters,
and then
also
a
row
column.
The
next
steps
so
we
can
continue
building
out.
Maybe
in
this
case
we're
exploring
the
parameters.
Now
we've
got
a
baseline
of
a
single
neural
network
and
maybe
we
want
to
see
how
this
compares
to
a
two-
layer
neural
network.
We
can
go
back
to
the
tuning
launch
which
was
just
minimized.
You
can
see
that
all
of
this
report,
the
graph
builder
report
as
well
as
the
table
are
just a
ppended
to
the
tuning
launch.
We
can
go
back
up
to t
his
tuning
launch
and
maybe
I
want
to
look
at
two
hidden
layers.
We'll
go
to
two,
and
now
I'll
put
in
a
range
of
those
results
or
those
parameters
and
keep
everything
else
the
same.
You
can
see
it's
going
to
preload
the
random
seed
that
I
generated
in
the
first
DOE
just
to
make
sure
everything's
using
the
same
random
seed.
We'll
keep
it
at
60,
even
though
we
might
want
to
increase
it
by
a
little
bit
since
we're a
dding
some
more
factors,
but
keep
it
simple,
we'll
just go
with
60.
One other
thing
just
to
mention
is,
in
the
case
it's
probably
pretty
low
likelihood t
hat
all
of
these
would
be
set
to
zero
in
a
design,
but
if
they
are
it'll
just
go
to
the
default
with
JMP,
which
is
a
single
10H
layer.
You
don't
have
to
worry
about
having
a
neural
network
with
zero
layers
or
zero
nodes.
If
we're
satisfied
with
this,
we're
just
going
to
click
Run.
While
this
is
running,
I'm
just
going
to
bring
up
the
home
window.
You
can
see
what
it's
doing.
We've
got
our
original
tuning
result
table
here,
and
it's
built
a
second t
uning
result
table,
and when
it's
done, i
t
just
appends
that
to
our
original
table.
Now
I
can
see
in
orange,
is
all
of
those
second-
layer
models.
We
can
see
in
this
case,
maybe
having
two
layers
was
not
very
beneficial
in
this
case.
Looks
like
in
pretty
much
all
cases,
those
neural
networks
are
not
as
good.
Then
maybe
just
to
round
this out,
we're
going
to
go
and buld some
neural
networks
with
boosting.
Boosting
is
a
really
nice
way
to
increase
the
prediction
power
of
our
model.
We're
going
to
specify
some
boosting
levels
from
1-10,
as
well
as
the
learning
rate
from
0.1
to
1.
Once
again
our
random
seed
is
the same,
and
these
are
going
to
take
a
little
bit
longer,
so
I'm
just
going
to
decrease the
number
of
runs
just
in
the
interest
of
time.
We
can
click
Run
here,
we
get
our
dialog
box
that
tells
us
our
overall
progress,
and
we
can
see
these
in
the
cases
where
we've
probably
got
lots
of
boosting,
they
might
be
taking
a
little
bit
of
time.
In
this
case
boosting
neural
networks,
or
neural
networks
that
had
boosting
are
now
this
purple,
and
we
can
see
in
pretty
much
all
the
cases
purple
is
higher,
especially
when
it
looks
like
we
can
see
by
these
box
plots
as
well.
Then
we
can
also
see
in t
his
case
over
here
our
boosting.
Maybe
we
haven't
found
the
maximum
number
of
boosts,
it
looks
like
it
continues
going
up,
so
we
might
want
to
add
some
additional
boosting
layers
or
maybe
our
learning
rate
is
best
at
this
middle
to
high
value
instead
of
at
the
lowest
learning
rate
values.
The
next
thing
we
might w
ant
to
do
once
we
have
this
initial
view
of
our
system,
is
to
run
a
few
of
these
neural
networks
and
compare
them.
We
can
do
that
a
few
ways.
We
can
look
in
our
table
here a
nd
run
or
and
select
some
neural
networks
based
on
the
graph,
and
you
can
see
they're
highlighting
down
below
in
our
data
table
as
well.
Maybe
what
we
want
to
do
is
run
the
top
four
or
five.
I
want
to
run
the
top
five.
I don't
have
to
go
back
into
the
neural
network
platform
to
do
this,
I
can
just
run
it
by
clicking
this
button,
run
selected
models.
Now
it's
going
to
run
those
back
into
the
neural
network
platform
and
give
us
a
little
output
that
shows
us
the
row
in
our
data
table
that
was
run.
This
is
just
the
standard
neural
network
output
now.
We
pass
through
the
same
random
seed,
so
if
we
take
a
look
at R-squared values,
we're
going
to
get
the
same
neural
network
that
was
developed
because
we're
using
that,
passing
through
that
same
seed.
Then
we
can
come
through
and
we
can
take
a
look
at more...
Either
the
more
summary
statistic
values
or
the
actual
versus
predicted,
or
get
the
profilers.
Whatever
we
might
want
to
do
to
compare
these
neural
networks.
One
other
thing
we
might
want
to
do
at
this
stage
is
create
some
additional
visualizations.
We
can
go
through
and
we
can
show
this
tuning
table.
If
we
don't
want
to
work
on i
t
here,
we
can
just
unhide
it,
and
now
we've
got
a
data
table
that
we
can
use,
and
we
can
go
into
graph
builder
and
maybe
we
want
to
take
a
look
at
validation
versus
training
R-
squared
just
to
see
how
that's
looking,
or
we
can
go
through.
Because
we've
got
the
number
of
DOE,
we can
track
how
our
neural
networks
have
improved
with
our
tuning.
I
can
go
into
graph
builder
and
I
can
use
that
nDOE,
and
maybe
we
want
to
look
at
training
and validation R-squared,
we
can
turn
on
box
plots
and
we
can
see
how
our
tuning
has
progressed
over
time
with
each
subsequent
design
that
we
ran.
Other
option
is
if
you
want
to
do
some
more
analytical
type
analysis
here,
we
can
go
into
screening
and
we
can
run
predictor
screening.
We
can
take
all
of
those
parameters,
load
them
in
as
predictors,
and
look at
our
training
and
validation
R-
squared.
Click
OK,
and
now
we
can
see
which
factors
might
be
most
influential
in the
training
or
validation.
Not
surprisingly,
those
learning
rates
and
boosting a
re
good
at
this
validation.
There's
no
end
to
what
analysis
you
might
want
to
do,
on
here
it's
just
what
your
goal
of
the
neural
network
was
and
where
you
want
to
go
with
it.
But
you
can
do
that
all
in t
his
data
table
which
now
has
160
neural
network
models.
Maybe
these
neural
network
models
take
a
little
bit
of
time,
so another
thing
you
can
do
is
we
can
just
save
this.
I
want
to
go
file,
and I
can
save
this.
I'm
just g
oing
to
save
it
to
my
desktop
as T
uning
Results.
I
can
come
back
to
this
later.
Now
that
I've
got
it
saved,
I'm
going
to
close
this.
One
thing
that
you'll
get
a
warning,
when
you
close
it.
You
should
just
hide
this
data
table.
If
you
close
it,
it's
going
to
take
it
out
of
the
neural
network
platform
and
you
won't
be
able
to
continue
your
work,
but
you
can
hide
it.
When
you
get
back
to
the
add-
in,
when
you
close
this
window,
it's
just
going
to
remind
you
.
You can
close
it
without
saving
or
you
can
cancel
to
go
back
and
save
it.
We're
just
going
to
click
OK.
When
I
save
this
tuning
table
for
later,
so
what
I
can
do
is,
is
now
I
can
relaunch
that
add-
in
and
I
can
go
into
the
tuning
add-
in,
click
OK.
But
instead
of
resetting
any
parameters
and
clicking
Run,
I
can
go
to
Load.
What
I
can
do
is
I
can
load
a
previously
saved
tuning
table,
and
what
this
does
is
let
me
in
the
case
of
neural
networks
that
take
a
long
time,
you
don't
have
to
start
over
from
scratch.
You
can
just
keep
building
on
the
table
that
you
wanted,
or
maybe
you
do
some
additional
analyses
offline
and
want
to
come
back
in
and
build
some
more
models,
so
you
can
go
through
and
when
you
specify
all
the
different
parameters
and
number
of
tuning
runs
you
want
to
have,
it's
just
going
to
continue
appending
it
to
this
table.
With
that
let's
minimize
this.
That's
quick
run
through
of a
ll
the
different
options
in
this
tuning
table.
It's
really
a
brute
force
way.
You
can
see
there's
no
auto
tune.
Right
now
it's
just
running
all
those
different
parameter
combinations
giving
you
an
idea
of where
your
optimal
model
might
be
.
How
you
might
work
this
into
your
workflow
so
you
can
explore
those
parameters,
this
is
essentially
what
we
did
today.
We
looked
at
single-
layer
models,
we
looked
at
models
with
two
layers.
Single-
layer
with
boosting.
Another
thing
you
might
do
is
go
back
and
say,
"W
ell,
where
are
my
optimal
models?"
They're
here.
I
can
go
back
to
the
tuning
launch
and
maybe
I
want
to
recreate
those
models
and
go
more
towards
this
range.
It
looks
like
I'm in
the
middle
range
here,
is
optimal,
so
instead
of
going
from
0-9,
maybe
I
go
from
5-7
.
Linear
models
also
might
be
best
in
the
middle,
and
maybe
the
gaussian,
there are
some
good
ones
on
the
left,
so
we'll
go
from
1-8
or
something
like
that,
seven
or
eight.
Then
we
can
look
at
which...
In
this
case,
there
wasn't
an
advantage
to
having
it
on
or
off,
so
we
could
just
have
those
off,
or
off
or
on.
But
it
looks
like
maybe
in
this
case
the
weight
decay
was
the
best
method,
so
we
could
just
include
that
one
and
then
go
through
and
build
some
more
models
with
a
narrow
range
of
those
parameters.
We
can
also
check
parameter
sensitivity
to
a
random
seed.
In
this
case
we
can
pass
through
either
constant
values
o
r
a
narrow
range.
You
can
pass
through
a
constant
set
of
parameters
just
by
setting
each
parameter
equal
to
each
other,
and
it'll
pass
that
through
the
neural
network
as
five
every
time,
or
you
can
just
specify
the
min
value,
and
that
will
pass
it
through
as
a
constant
value.
If
we
wanted
to
we
could
look
at
maybe this
was
a
neural
network
that
was
one
that
we
identified
as
optimal,
but
I
want
to
see
how
robust
it
is
to
the
random
seed.
What
we
can
do
is
we
can
just
say
just
run
it
once,
but
let's
replicate
it
with
20
random
seeds.
That
might
actually
take
a
little
bit
of
time,
so
let's
go
down
to
10.
Then
we
can...
Actually,
I
don't
want
to
do
that
on
this
one,
because
I've
got
all
those
others,
but
we'll
just
run
this.
You
might
want
to
do
this
on
a
clean
data
table
where
you're
not
adding
it
to
the
design,
but
we
can
use b
ecause
we
have
the
number
of
DOEs
we
could
find
where
that
random
seed
was.
We
can
see
here,
all
the
different
random
seeds
and
I
can
go
into
graph
builder
and
I
could
just
put
in
the
training
and
validation
R-
squareds,
and
then
I
can
use
local
data
filter
now
on
my
random
seed
to
show
everything
except
the
one
where
we
had
all
those.
Now
I
can
see
how
robust
my
model
is
to
that
random
seed.
That's
the
same
design
now
run
10
times,
and
so
on
average
we
can
see
what
the
median
value
is
or
the
range
of
expected
results.
Then
finally,
w
e
can
load
those
saved
data
tables.
Those
are
really
beneficial
for
neural
networks that might have
really
long
computing
times,
so
you
can
save
it
off
and
come
back
to
it
later.
Those
are
some
workflows
that
might b
e
helpful
to
you
as
you
build
your
neural
network
models.
I
just
want
to
wrap
up
with
a
few
lessons
learned.
This
was
the
first
scripting
project
that
I
went
through
that
developed
an
application,
and
so
I
think
this
is
pretty
obvious
to
anybody
that's
developed
any JSL
or
built
any
applications,
the
JMP
community
is
really
an
invaluable
resource.
I
would
say
95%
of
all
my
questions,
I
would
just
go
to
the
community
and
someone
else
had
a
similar
issue.
Really
recommend
if
you
are
interested
in
building
add-
ins,
building
applications,
the
community
is
really
the
first
place
to
go.
Really,
your
imagination
and
your
targets
are
the
limiting
factor
in
developing
these
applications.
A
ligned
with
that is
because
there are
so
many
possibilities,
there
was
a
need
to
keep
this
focused.
In
this
case,
I
just
have...
I
think
in
one
iteration
I
had
many
different
outputs,
different
graphs,
but
they
weren't
really
the
heart
of
the
application.
Because
there's
so
many
possibilities,
I really
tried
to
keep
this
focused
on
just
giving
some
graphical
output
to
navigate
the
different
models
and
run
them
and
identify
which
might
be
most
appropriate.
What's
next?
I
really
want
feedback.
It's
selfish,
but
it's
what
I
really
want.
I
would
love
to
hear
how
people
can
use
this,
how
it's
benefiting
them.
You
can
find
this
in
the
community,
in
the
add-
in
section
of
the
community.
Please
feel
free
to
make
a
post
here,
tell
me
how
you're
using
it.
I
do
have
a
few
known
issues,
and
as
I
gather
more,
I'm
sure
it's
not
100%
bug-
proof,
but
this
will
be
the
place
to
go
to
get
the
current
version
of
the
add-in
as
well
as
understand
any
known
issues.
But
I'd
love
to
hear
how
you're
using
it,
and
the
community
is
a
great
place
to
do
that.
I
would
also
like
to
hear
ideas
on
additional
reports.
I
kept
this
pretty
simple,
but
if
you
find
that
there's
a
report
that
you're
generating
every
time
you
run
this,
I'd
like
to
know
what
that
is a
nd
maybe
we
can
build
that
into
the
add-
in.
I'm
also
going
to
continue
debugging.
There
are
a
few
known
issues
that
I'm
working
through
right
now,
and
then
maybe
a
larger
goal
is
to
try
to
do
that
auto
tune.
Instead
of
doing
this
brute
force
method,
maybe
start
out
with
a small
number o
f
parameters,
figure
out
which
ones
are a
ffecting
the
response
the
most,
and
then
having
a
directed
tuning.
With
that,
I'd
like
to
thank you
for
listening
to
this
presentation.
This
is
a
really
fun
project
and
I
hope
you
find
it
useful,
and
would
love
to
hear
about
it.
Thanks.
