Hi,
everyone.
Thanks
for
coming
to
our
video.
My
name
is
Ryan
Parker,
and
today
I'm
going
to
present
with
Clay
Barker
about
some
new
tools
that
we
have
added
to
analyze
Spectral
Data
with
the
Functional
Data
Explorer
in
JMP
Pro
17.
First,
I
just
wanted
to
start
off
with
some
of
the
motivating
data
sets
that
led
us
to
add
these
new
tools.
They're
really
motivated
by
these
chemometric
applications,
can
definitely
be
applied
to
other
areas,
but
for
example,
we
have
this
spectroscopy
data
where
the
first
thing
you
might
notice
with
this
is
we've
got
a
lot
of
data
points
sampled,
but
we
also
have
some
very
sharp
peaks
in
our
data.
That's
going
to
be
a
recurring
theme
where
we
have
a
need
to
really
identify
these
sharp
features
that
the
existing
tools
we
have
in
JMP
are
a
little
difficult
to
really
capture
those.
For
example,
we're
thinking
about
composition
of
materials
or
how
we
can
detect
biomarkers
and
data.
These
are
three
spectroscopic
examples
that
we'll
look
at.
Another
example
of
data
that
is
of
interest
is
this
mass
spectrometry
data.
Here
we're
thinking
about
a
mass
to
charge
ratio
that
we
can
use
to
construct
a
spectrum
where
the
peaks
in
the
spectrum,
they're
representing
proteins
that
are
of
interest
in
the
area
of
application.
One
example
is
comparing
these
spectrums
between
different
patients,
say
a
patient
with
cancer
or
a
patient
without,
and
the
location
of
these
proteins
is
very
important
to
identify
differences
in
these
two
groups.
Another
example
is
chromatography
data.
Here
we
can
think
about
we're
using
some
chemical
mixture
over,
we're
using
a
material
that's
going
to
help
us
quantify
relative
amounts
of
the
various
components
that
are
in
these
mixtures.
By
using
the
retention
time
in
this
process,
we
can
try
to
identify
the
different
components.
For
example,
if
you
didn't
know
this,
I
was
not
aware
until
I
started
to
work
with
this
data,
trying
to
impersonate
olive
oil
is
a
big
deal.
We
can
use
these
data
sets
to
figure
out,
okay,
what's
a
true
olive
oil,
or
what's
just
some
other
vegetable
oil
that
someone
might
be
trying
to
pass
off
as
an
olive
oil?
The
first
thing
I
want
to
do
is
go
through
some
of
the
new
preprocessing
options
that
we've
added
to
help
work
with
spectral
data
before
we
get
to
the
modeling
stage.
We
have
a
new
tool
called
the
Standard
normal
Variate,
a
multiplicative
scatter
correction
where
you
have
if
you
have
light
scatter
in
your
data,
the
SavitzkyâGolay filter,
which
is
the
smoothing
step
when
you
have
spectral
data
that
we'll
get
into.
Finally,
a
new
tool
to
perform
a
baseline
correction
for
the
data
to
remove
trends
that
you're
not
really
interested
in
that
you
want
to
get
out
first.
Okay,
so
what's
standard
normal
variant?
Currently
in
JMP,
we
have
the
ability
to
just
standardize
your
data
in
FDE.
But
when
you
use
that
tool,
it's
just
taking
the
mean
of
all
of
the
functions
and
a
global
variance
and
scaling
it
that
way,
but
with
standard
normal
variant,
we're
thinking
about
the
individual
means
and
variances
of
each
of
the
functions
to
standardize
and
remove
those
effects
before
we
go
to
analysis.
Whenever
I'm
on
the
right
here,
after
performing
the
standard
normal
variant,
we
can
see,
okay,
there
was
some
overall
means
and
now
they're
all
together
and
any
excess
variance
is
taken
out
before
we
go
to
analysis.
Multiplicate
of
scatter
correction
is
the
next
step,
and
it's
an
alternative
to
standard
normal
variant.
In
some
cases
you're
thinking
you're
going
to...
Whenever
you
use
it,
you
may
end
up
with
similar
results.
The
difference
here
is
the
motivation
for
using
multiplicate
of
scatter
correction.
That's
when
you
have
light
scatter,
or
you
think
you
might
have
light
scatter
because
of
the
way
that
you
collected
the
data.
What
happens
is
for
every
function,
we're
going
to
fit
this
simple
linear
model
where
we've
got
a
slope
and
an
intercept,
and
we're
going
to
use
those
estimated
coefficients
to
now
standardize
our
data
that
we're
going
to
work
with.
We
subtract
off
the
intercept
and
divide
by
the
slope,
and
now
we
have
the
standardized
version.
A
gain,
you
can
end
up
with
similar
results
as
standard
normal
variance.
Now
the
next
preprocessing
step
I'm
going
to
cover
is
the
SavitzkyâGolay filter.
When
you
have
spectral
data,
before
we
get
to
the
modeling
stage,
the
new
modeling
tools
we
have,
they're
developed
in
such
a
way
that
they're
trying
to
pick
up
all
the
important
pieces
of
the
data.
If
you
have
noise,
we
need
to
do
a
step
where
we
smooth
that
first.
That's
where
the
SavitzkyâGolay filter
comes
in.
What
we're
doing
is
we're
going
to
fit
an
end-
degree
polynovial
over
a
specified
bandwidth
that
we
can
choose
to
help
try
to
remove
any
noise
from
the
data.
In
FDE,
currently,
we're
going
to
go
ahead
and
select
those
best
parameters
for
you,
the
degree
and
the
width
to
try
to
minimize
the
model
error
that
we
get.
One
thing
I
do
want
to
point
out
is
that
we
do
require
a
regular
grid
to
give us
this
operation,
which
will
come
up
again
in
the
future,
but
FDE
is
going
to
create
one
for
you.
We
also
have
this
reduced
grid
option
available
if
you
want
finer
control
first
before
you
rely
on
us
making
that
choice
for
you.
The
nice
thing
about
this
SavitzkyâGolay filter
is
because
of
the
way
the
model
is
fit,
we
now
have
access
to
derivatives.
This
is
something
that
has
come
up
prior
to
spectral
data,
and
now
that
we
have
this,
we've
got
a
nice
way
for
you
to
access
and
work
with
modeling
these
derivative
functions.
The
last
one
I
want
to
cover
is
the
baseline
correction.
What
baseline
correction
is
doing
is
it's
thinking
about
there
might
be
overall
trends
in
our
data
that
we
want
to
get
rid
of.
This
data
set
on
the
right
has
just
a
very
small
differences
like
linear
difference
in
the
functions.
What
we're
thinking
about
is,
okay,
we
don't
really
want
to
care
about
that,
we
want
to
get
rid
of
it.
What
this
tool
is
going
to
allow
you
to
do
is
select
the
baseline
model
that
you
want. I
n
this
case,
it's
just
a
really
simple
linear
model,
but
you
may
have
some
where
you've
got
exponential
or
logarithmic
trends
that
you
want
to
get
rid
of.
And
so
we
have
that
available.
Then
you
can
select
your
correction
region.
For
the
most
part,
you're
going
to
want
to
correct
the
entire
function,
but
it
may
be
possible
that
maybe
only
the
beginning
or
the
end
of
the
function
is
where
you
want
to
correct.
We
end
up
with
these
baseline
regions
that
are
these
blue
lines.
If
we
click
this
add
button,
it'll
give
us
a
pair
of
blue
lines.
We're
going
to
drag
these
around
to
parts
of
the
function
that
we
believe
are
real.
All
the
peaks
in
these
data
is
something
that
we
don't
really
want
to
touch.
This
is
the
part
of
the
functions
that
we
want
to
keep
and
analyze
and
is
going
to
give
the
information
that
we're
interested
in.
Also
if
you
select
this
within
region,
anything
that's
within
these
regions
is
what
will
get
corrected.
You're
either
going
to
do
one
or
the
other,
right?
You're
going
to
either
leave
it
alone
or
you're
going
to
change
only
within
that
in
those
data
sets.
Finally,
you
don't
see
it
here,
but
you
can
also
add
anchor
points.
It
may
be,
depending
on
your
data,
easy
to
just
specify
a
few
points
that
you
know
this
describes
the
overall
trend.
When
you
click
add,
you'll
get
a
red
line,
and
that's
going
to
tell
you
this,
wherever
I
drag
that
line,
it's
definitely
going
to
be
included
in
the
model
before
I
correct
the
baseline.
When
you
click
okay
here,
you'll
just
end
up
with
a
new
data
set
that
has
the
trend
removed.
Okay,
so
that
brings
us
to
the
modeling
stage.
What
we've
added
for
JMP
Pro
17
are
wavelet
models.
Okay,
so
what
are
wavelet
models?
They
are
basis
function
models,
not
like
anything
we
have
currently
in
JMP,
but
they
can
have
very
dramatic
features.
What
these
features
are
doing
are
helping
us
pick
up
these
sharp
peaks
or
these
large
changes
in
the
function.
We
also
have
the
simple
Haar
wavelet,
which
is
just
a
step
function.
If
it
turns
out
that
something
really
simple
is
like
the
step
function
fits
best,
we
will
give
you
that
as
well.
You
can
see
we
have
a
few
different
options
that
are
really...
If
you
think
about
bending
these
wavelets
and
stretching
them
out,
that's
how
we
are
modeling
the
data
to
really
pick
up
all
these
features
of
interest.
To
just
motivate
that,
I
want
to
show
you
the
current
go-
to
in
JMP,
which
is
a
B-
spline
model,
which
has
a
very
difficult
time
picking
up
on
these
features
without
any
hand-
tuning.
B-
spline
model
is
doing
a
little
bit
better.
It
still
has
some
issues
picking
up
the
peaks,
but
it
might
in
some
ways
be
the
best.
Direct
functional
PCA,
doing
almost
as
good
as
P-sp
lines,
but
not
quite.
Then
we
have
wavelets.
We're
really
picking
up
the
peaks
the
best.
In
this
particular
data
set,
it's
not
fitting
them
perfectly,
but
we
would
think
looking
at
diagnostics,
the
wavelet
model
is
definitely
the
one
we
would
want
to
go
with.
Again,
we
have
these
five
different
wavelet
model
types,
and
what
we're
going
to
do
is
we'll
fit
all
these
for
you
so
that
you
don't
have
to
worry
about
picking
and
choosing.
Outside
of
the
Haar wavelets,
all
of
the
other
wavelet
types
have
a
parameter.
We
have
a
grid
that
we
are
going
to
search
over
for
you
in
addition
to
the
type.
Now
it
may
be
possible
that
there
are
some
cases
where
users
have
said,
hey,
this
particular
wavelet
type
is
exactly
how
my
data
should
be
represented,
so
you
can
change
the
default
model,
but
by
default,
we're
going
to
pick
the
model
that's
going
to
optimize
this
model
selection
criteria,
the
AISC.
Really
what
you
can
think
about
here
is
there
could
be
potentially
a
lot
of
parameters
in
every
one
of
these
wavelet
models.
We're
effectively
using
a
Lasso
model
to
try
to
remove
out
any
parameters
that
really
just
aren't
fitting
the
data.
We
get
a
sparse
representation,
no
matter
the
wavelet
model.
We
saw
this
earlier
where
we
have
to
have
our
data
on
the
grid.
It's
the
same
thing
with
wavelets.
If
you
just
start
going
through
the
wavelet
models
and
your
data
are
not
on
the
grid,
we'll
create
one
for
you.
But
again,
just
wanted
to
point
out
you
can
use
that
reduce
grid
option
to
have
finer
control.
Okay,
so
something
else
that
we
show
that
can
help
give
you
some
insight
into
how
these
models
work
is
this
coefficient
plot.
What
it's
telling
us
is
this
X
axis
is
the
normal
X
of
the
input
space
of
your
function,
but
the
Y
axis
is
the
resolution.
These
top
resolutions
here,
you're
thinking
about
overall
means.
As
we
get
into
these
high
resolutions,
these
are
the
things
that
are
happening
really
close
together.
A
red
line
means
it's
a
negative
coefficient.
Blue
means
it's
positive.
They're
scaled
so
that
they're
all
interpretable
against
each
other.
The
largest
lines
give
you
the
idea
where
the
largest
coefficients
are.
We
can
see
that
the
higher
frequency
items
are
really
here
at
the
end
of
the
function.
We
have
some
overall
trends,
but
just
something
to
think
about
that
these
wavelet
models
are
looking
at
different
resolutions
of
your
data.
Something
else
that
we've
added
before
we
get
to
our
demo
with
Clay
is
wavelets
DOE.
In
FDE,
we
have
a
functional
DOE
that
is
working
with
functional
principal
components.
If
you
don't
know
those
are,
that's
okay.
All
you
need
to
know
is
that
with
wavelets,
we
have
coefficients
for
all
of
these
wavelet
functions.
In
this
DOE
analysis,
we're
thinking
about
modeling
the
coefficients
directly.
The
resolution
tells
you
an
idea
of
if
it's
a
high-
frequency
item
or
low-
frequency
item.
Then
this
number
in
the
brackets
is
telling
you
the
location.
You
can
think,
okay,
these
items
here
are
in
the
threes,
and
that's
where
some
of
that
highest
features
were
that
we
saw
in
that
coefficient
plot.
Those
have
what
we're
calling
the
highest
energy.
Energy
in
this
case
is
just...
If
we
score
all
the
coefficients,
we
add
them
up,
you
can
think
of
that
being
as
the
total
energy.
So
this
energy
number
here
is
a
relative
energy
and
giving
you
an
idea
of
how
much
energy
it
is
explaining
in
the
data.
The
nice
thing
really
about
using
the
coefficient
approach
is
these
have
direct
interpretation
right
to
the
location
and
to
the
resolution.
An
alternative
that
you
can
try
and
compare
against
functional
PCA
or
functional
DOE
if
you
have
this
interpretability
of
the
coefficients.
Now
I
think
I'll
hand
it
over
to
Clay.
He's
got
a
demo
for
you
to
see
how
you
use
these
models
in
JMP
Pro.
Thanks,
Ryan.
Let's
take
a
look
at
an
example
that
we
found.
Ryan
mentioned
briefly
the
olive
oil
data
set
that
we
found.
It's
a
sample
of
120
different
oils.
Most
of
them
are
olive
oils,
some
of
them
are
blends
or
vegetable
oils.
What
we
wanted
to
see
is,
can
we
use
this
high-
performance
liquid
chromatography
data?
Can
we
use
that
information
to
classify
the
oil?
Can
we
look
at
the
spectraland
say
this
is
an
olive
oil
or
this
is
not
an
olive
oil?
These
data
came
out
of
a
study
from
a
university
in
Spain,
and
Ryan
and
I
learned
a
lot
about
olive
oil
in
the
process.
For
example,
olive
oil
is
actually
a
fruit
juice,
which
I
did
not
know.
Let's
take
a
look
at
our
data.
Each
row
in
our
data
set
is
a
different
olive
oil
or
other
oil,
and
the
rows
represent
the
spectra.
We'll
use
the
Functional
Data
Explorer,
and
it'll
take
just
a
second
to
fit
the
wavelet
models.
Y
ou'll
see
here,
we
fit
our
different
wavelets.
As
Ryan
mentioned
earlier,
we
try
a
handful
of
different
wavelets
and
we
give
you
the
best
one.
In
this
case,
the
Simlet
20
was
the
best
wavelet
in
terms
of
how
well
it
fits
our
data.
We
can
see
here
where
we've
overlaid
these
fitted
wavelets
with
the
data
that
this
wavelet
model
fits
really
well.
L
et's
say
you
had
a
preferred
wavelet
function
that
you
wanted
to
use
instead,
you
can
always
click
around
in
this
report
and
it'll
update
which
wavelet
we're
using.
If
we
wanted
the
Simlet
10
instead,
all
you
have
to
do
is
click
on
this
row
in
the
table,
and
we'll
switch
to
the
Simlet
10
instead.
Let's
go
back
to
the
20
and
we'll
take
a
look
at
our
coefficients.
In
the
wavelet
report,
we
have
this
table
of
wavelet
coefficients.
As
Ryan
was
saying
earlier,
these
give
us
information
about
where
the
peaks
are
in
the
data.
The
further
wavelet,
we
think
about
that
like
an
intercept,
so
that's
like
an
overall
mean.
Then
every
one
of
these
wavelet
coefficients
with
a
resolution
is...
It
lines
up
with
a
different
part
of
the
function.
This
resolution
one
is
the
lowest
frequency
resolution,
and
it
goes
all
the
way
up
to
resolution
12.
These
are
much
higher
frequency
resolutions.
As
you
can
see,
we've
zeroed
a
lot
of
these
out.
In
fact,
this
whole
block
of
wavelet
coefficients
is
zeroed
out.
That
just
goes
to
show
that
we're
smoothing.
If
we
used
all
of
these
resolutions,
it
would
recreate
the
function
perfectly,
but
we
zero
them
out
and
that
gives
us
a
much
smoother
function.
We
fit
the
wavelet
model
to
our
spectral
and
we
think
we
have
a
good
model.
Let's
take
these
coefficients
and
we're
going
to
use
these
to
predict
whether
or
not
an
oil
is
olive
oil.
I've
got
that
in
a
different
data
set.
Now
I've
imported
all
of
those
wavelet
coefficients
into
a
new
data
set
and
I've
combined
it
with
what
type
of
oil
it
is.
It's
either
olive
oil
or
it's
other,
and
we've
got
all
of
these
wavelet
coefficients
that
we're
going
to
use
to
predict
that.
The
way
we
do
that
is
using
the
generalized
regression
platform.
We're
going
to
model
the
type
using
all
of
our
different
wavelet
coefficients.
Since
it's
a
binary
response,
we
choose
the
binomial
distribution,
and
we're
interested
in
modeling
the
probability
that
an
oil
is
olive
oil.
Because
we
don't
want
to
use
all
of
those
wavelet
coefficients,
we're
going
to
use
the
Lasso
to
do
variable
selection.
Now
we've
used
the
Lasso
and
we've
got
a
model
with
just
14
parameters.
Of
all
of
those
wavelet
coefficients
that
we
considered
for
our
model,
we
only
really
needed
14
of
them.
We
can
take
a
look,
we've
zeroed
out
a
lot
of
those
wavelet
coefficients.
Let's
take
a
look
at
the
confusion
matrix.
Using
our
model,
we
actually
perfectly
predicted
whether
or
not
one
of
these
oils
is
an
olive
oil
or
it's
something
else.
That's
pretty
good.
We
took
our
wavelet
coefficients
and
we
selected
the
13
most
important
because
one
of
those
14
parameters
is
the
intercept.
We
only
needed
13
of
those
wavelet
coefficients
to
predict
which
oil
we
had.
In
fact,
we
can
take
a
look
at
where
those
wavelet
coefficients
fall
on
our
function.
What
we
have
here
is
we
have
the
average
olive
oil
spectralin
blue
and
the
other
oils
in
red,
and
each
of
those
dashed
lines
lines
up
with
the
coefficients
that
we
used.
Some
of
these
really
make
a
lot
of
sense.
For
example,
here's
one
of
the
wavelet
coefficients
that
is
important,
and
you
can
see
that
there's
a
big
difference
in
the
olive
oil
trace
and
the
other
oil.
Likewise,
over
here,
we
can
see
that
there's
a
big
difference
between
the
two
there.
You
can
look
through
and
see
that
a
lot
of
these
locations
really
do
make
sense.
It
makes
sense
that
we
can
use
that
part
of
the
curve
to
discriminate
between
the
different
types
of
oil.
We
just
thought
that
was
a
really
cool
example
of
using
wavelets
to
predict
something
else.
Not
that
olive
oil
isn't
fun,
but
Ryan
and
I
both
have
young
kids
and
we're
both
big
fans
of
the
world.
We
also
found
a
new
world
data
set
where
someone
had
recorded
wait
times
for
one
of
the
popular
rides
at
Disney
World.
It's
called
the
Seven
Dwarfs
Mind
Train
.
It's
a
roller
coaster
at
Disney
World.
Someone
had
recorded
wait
times
throughout
the
day
for
several
years
worth
of
data.
I
also
mentioned
these
are
a
subset
of
the
data.
One
of
the
problems
is
the
parks
are
open
for
different
amounts
of
time
each
day,
and
some
of
the
observations
are
missing.
We
subset
it
down
and
got
it
to
a
more
manageable
data
set.
I
would
say
that
this
example
is
inspired
by
real
data,
but
it's
not
exactly
real
data
once
we
massaged
it
a
little
bit.
If
we
graph
our
data,
we
can
see...
The
horizontal
axis
here
is
the
time
of
day,
and
the
vertical
axis
is
the
wait
time.
In
the
middle
of
the
day,
the
wait
time
for
this
ride
tends
to
be
the
highest.
We
can
look
around
at
different
days
of
the
week.
Sunday,
Monday
is
a
little
bit
more
busy.
Tuesday
is
a
little
less
busy,
Saturday
is
the
most
busy.
We
can
do
the
same
thing
looking
at
the
years.
This
is
2015,
2016,
2017.
It
looks
like
every
year
it's
getting
longer
and
longer
wait
times
until
something
happens
in
2021.
I
think
we
all
know
why
wait
times
at
an
amusement
park
would
be
less
in
2021.
We've
got
an
idea
that
you
can
use
this
information,
like
day
of
the
week,
year
and
month,
to
predict
what
that
wait
time
curve
will
look
like.
Let's
see
how
we
do
that
in
FDE.
I'll
just
run
my
script
here.
What
we've
done
is
we'll
come
to
the
menu
and
ask
to
fit
our
wavelet
model.
It
takes
just
a
second,
but
really
not
that
long
to
fit
several
years
worth
of
data.
This
time
we're
not
using
the
Simlet
anymore.
We're
using
this
Daubechies
wavelet
function.
What
Ryan
mentioned
earlier
was
the
wavelet
DOE
feature.
Now,
what
I
didn't
show
was
that
we've
also
loaded
time
of
the
day
of
the
week
and
the
year
and
the
month
in
the
FDE.
We're
going
to
use
those
variables
to
predict
the
wavelet
coefficients.
Let's
go
to
the
red
triangle
menu
and
we'll
ask
for
wavelet
DOE.
Now,
what
is
happening
behind
the
scenes
is
we're
using
day
of
the
week,
month,
and
year
to
predict
those
wavelet
coefficients,
and
then
we
put
it
all
back
together
so
that
we
can
see
how
the
predicted
wait
time
changes
as
a
function
of
those
supplementary
variables.
Now,
of
course,
we
summarize
it
in
a
nice
profiler.
We
can
really
quickly
see
the
effects
of
month.
If
we're
just
going
by
the
average
wait
time
of
this
particular
ride,
we
can
see
that
September
tends
to
have
the
lowest
wait
time.
We
can
really
quickly
see
the
COVID
effect.
The
wait
times
were
here
in
2019,
and
then
when
we
went
forward
to
2020,
they
really
dropped.
You
can
look
around
to
see
which
day
of
the
week
tends
to
be
less
busy,
which
months
are less
busy. I
t's
really
a
cool
way
to
look
at
how
these
wait
times
change
as
a
function
of
different
factors.
Thank
you
for
watching.
That's
all
we
have for
today
and
we
hope
you'll
give
the
wavelet
features
in
FDE
a
try.
Thanks.
