Converting Binary Responses to Continuous in Product Development Using DOE - (2023-US-30MP-1464)

cpark2

At HP Hood, the use of design of experiments (DOE) has helped to successfully identify formulations in new product development, delighting consumers of our food products.

Throughout our R&D department's wide implementation of DOE, there have been a number of successful DOE models, as well as some unsuccessful ones. One major stumbling block to model building with DOE has been dealing with subjective binary responses such as acceptable/unacceptable. These binary responses provide less information than continuous responses, thus inhibiting the ability to extract meaningful results from designed experiments.

This presentation shares simple and practical strategies for using the JMP DOE platform to convert these binary responses to continuous ones, resulting in improved models and powerful insights. Real-world examples from consumer food products are given to demonstrate how DOE can be used for more than just building models. It can be used to overcome the problem of responses that are difficult to measure.

This talk is titled Expanded Uses of Converting Binary Responses

to Continuous Responses in Consumer Product Development.

It's a bit of a mouthful,

but I promise it won't be that complicated.

My name is Curtis Park. I'm a principal scientist at HP Hood.

HP Hood is a company, a food and beverage company.

We make a lot of different milks, nondairy milks.

We also make yogurt, cottage cheese, ice cream.

So a lot of a lot of fun things to taste at work.

I'm a food scientist by education.

A few years ago I was asked to take a look at a problem

that we had for one of the beverages that we were producing.

I'm going to show you a video just so you can see.

But we were getting a lot of consumer complaints

and these complaints were happening

when the product was close to the end of shelf life.

A s you see in this video,

it's pretty obvious why people were complaining.

I think I would complain if I saw something like that too.

It's supposed to be a nice portable beverage.

It's thick and chunky when it's being poured out.

Not what I would expect.

Believe it or not, this product was not spoiled.

I promise you, it was not spoiled.

So I was asked to take a look at this and figure out how can we fix it?

What's the problem? How do we fix it?

HP Hood at the time, this was a few years ago.

We were early on in our journey with using JMP,

and so I was really excited to have an application to use in real life

rather than just reading about it or learning about it.

Naturally I felt like this, like Yahoo! Let's run a DoE, let's do it.

I was really excited

and for those of you who might not have as much experience doing DoE,

the first step is usually taking a look at what factors should I be looking at.

So we did a few experiments.

If you can forgive me, they were probably one factor at a time experiments.

But we narrowed in on what we believed were the key ingredients

that could have been causing the problem.

We ended up making a design.

This is probably the fourth or fifth iteration of the design

that we came up with,

and this was in custom design.

So if you go to custom design,

that's that's the platform that we use to generate this DoE .

A s you can see, this is this is what we had.

So we had ingredients A, B and C, and it was actually a response surface.

So we had all of the two way interactions

and the quadratic terms built into the model.

It ended up being 17 runs, as you can see here.

It's 17 different treatment combinations.

This much A , this much B , this much C for each run.

Once we've settled on this design,

we were really excited so let's go solve this problem.

Piece of cake, right?

You go into the lab, into our pilot plant,

you throw some things together, the beverage comes out.

I'm making it a lot more simple than it actually is.

We made 17 different beverages

and then we put them on the shelf for a little while

because as I mentioned earlier,

it takes a little bit of time for this problem to appear.

Put them on the shelf for a while, sat until they were ready to be analyzed.

This is just a screenshot of a data table.

This has our actual or our design that we used.

A s you can see, there's a column here to the right that I highlighted.

It's our our friend, the Y, our response column.

So once we got to the point where we were ready to ready to measure that chunky pour

now we started thinking, Oh, how are we going to measure that?

Because a t the time, we did not have a chunky parameter.

I've never heard of one. I've never found one.

If anyone has ever found one, we'd love to to see it and maybe buy one.

But it's our knowledge. It doesn't exist.

So what options did we have to measure this?

Because if you can't measure it with DoE it's really not that useful.

So we have a fe w options.

First thing is we can measure everything as a binary response.

So it's either a pass fail, it's good or bad etc.

There's some pros with this and some cons.

The pros would be it's pretty simple to do, right?

Anybody can say pass or fail and it takes you like no time to to measure it.

However, it has some serious cons to it.

Such as, it's really subjective to the observer.

What I think is good,

a colleague of mine might think is bad.

Or even worse, what I think is good, my boss might think is bad.

So it's really subjective.

While it can give you some information,

they don't give us as much information as we want.

Because when you do logistic regression, what you get out of it really are just

probabilities of something passing or probabilities of failing.

In my experience, that's been difficult to communicate

and to really understand what to do with that data,

especially when we're trying to communicate with non-technical people.

So continuous if there's any way to get a continuous response,

that's what we strive for because they give us a lot more information.

We can know how good is it or how bad is it,

because not all good are created equal.

There's another option we could have done

and I would say this is probably the best option

if you can do it, is we could run consumer testing and get consumer input.

What this would look like is I have all our beverages, 17 beverages,

and we recruit maybe 100, 120 consumers of our product

and we have them sit down and rate every single one

for different attributes,

one of them probably being how well do you like how this pours?

The reason why this is a gold standard

is because those are the people's opinions who matter to us.

What we would do is after we get 100 or 120 responses,

you take a look at the data you get,

we can take averages and put those averages into our model.

However,

it can cost a lot of money and it can take a lot of time.

So if your budget doesn't allow it

or your timeline for whatever reason doesn't allow it,

you can't do this for everything.

Sometimes the thing you're trying to measure

isn't such a huge problem that you're trying to solve

that it's worth spending all that money.

But it would still be important to be able to measure it.

Do you have any other options?

I mentioned this earlier.

You can find an instrument that can measure what you're looking for.

Sometimes they exist.

Like I said, I don't know of a chunky parameter.

I looked in our warehouse in our R&D center, couldn't find one.

Even if you can find one,

if this is something that's really specialized,

you're not going to use it very often.

It doesn't make sense to buy the piece of equipment or it could be something

that would be really great, but it requires a lot of expertise

that maybe your R&D, your technical department doesn't have or

just doesn't have the time or resources to to deal with.

I'm going to show you the last option we have here.

What I'm going to say is training a group of people how to rate that attribute

of interest and then let them give you all the ratings.

This is quite as good as having actual consumers.

But here we're trying to take subjectivity out of it

and make it objective.

When well trained humans can be great measuring instruments.

I'm going to walk you through what we've done at Hood

when we have some hard to measure attribute.

We're going to use the case study of this chunky pour.

This is our roadmap.

I'll walk you through this and then we'll actually do it live.

The first thing I wanted to get across

is that the samples that you produce from DoE can be used for many purposes.

I like to tell people that your samples are like gold

and you should treat them like gold.

They're very valuable.

You may do a DoE thinking that you're trying to answer one question,

but something else might pop up later

that you would be able to use those samples to answer that question as well.

I've had that happen to me many times, so sometimes it's good to think about

just ask yourself the question.

I've done all this work to make 17 different beverages.

What else can I do with them?

What else can I learn?

In our case, we use these samples as a "calibration set"

so that we can teach our humans, my colleagues,

how to measure this chunky pour.

So here's our method.

The first thing we do is we review

all the samples with a small group, some maybe 1 or 2 or 3 people that are

really knowledgeable on the subject or are responsible for the project.

What you do is you look at all the samples

and decide which samples should be used to train the Raiders.

We're trying to build a scale essentially, and then we'll take that scale

and we'll get our friends, let's say 10, 15, 20 friends to actually rate these,

these samples for us after we've trained them.

Training step two,

have them read each video, step three .

If it's a video, it could be something else, a picture,

or it could be actually them pouring out the product

if you have enough, etc.

You can get the idea.

Next, we'll take the average of all those ratings.

We'll look at the data, make sure there's nothing funky in there

and then we will use those average values to build a model.

Let's start with, oops.

Let's start with steps one and two.

So we're going t o assume that we've looked at all the all the videos

and the way we typically do it because it's a little easier

is you start off answering the question, which one is the lowest in Chunky pour?

That would be this one right here.

Number one, I'm going to play each one of these.

This just to make it clear, this is our scale.

It's a continuous scale from 1 to 10 and the 1 to 10 is kind of arbitrary.

If if you have something that works better for you then great.

The video right above it corresponds to that.

So this first video corresponds to a one.

So as you can see, while we're watching this video

pours nicely, no rippling and no chunkiness.

Pours as expected. Beautiful.

That's that's the easy sample to identify

and then in the in the sample set, we ask ourselves, okay, which one is the worst?

In this case, it was pretty obvious.

I will tell you again, this product is not spoiled.

So just with changing a few ingredients.

You can see it's so thick, we can't even get it out of the bottom.

So that's obviously a 10.

Then we did a little bit of work

to try to figure out, okay, which one should we consider to be a five?

So halfway in between.

This one, you can see it still flows, but there is chunkiness to it.

Then maybe a two and a half would be this one.

See it has a little less chunkiness to it.

Flows well, probably with normal shaking.

It'd probably be fine.

So there's a little bit of subjectivity,

but you add more people to make it more objective.

Then the last one.

This is seven and a half.

So you can see it's very, very chunky.

The only thing that really is differentiating it from number ten is

that we can get it out of the bottle still flows.

But as you can see, it's pretty thick.

What I would do and basically in this amount of time,

I could train the people that are going to help us

to analyze this, to measure this chunky pour.

Then we'll have them rate once we've trained them.

I'll basically do what I just did.

Maybe we'd take a little bit more time

to be more specific with certain things we want them to be looking for.

If what you're having someone rate is a lot more complicated,

then you'll probably have to need to take more time training people.

This one wasn't pretty complicated

and we're really just looking for people's first impression.

A fter that you have them rate all the videos

i like to use Microsoft forms just because it's easy and I can get the the data

really quickly and easily, but you can use whatever you want,

including paper, although that takes more time and I try to avoid that.

Just to show you what Microsoft, what our forms look like.

Here's a preview of it.

This is as if you're doing it on your phone.

I like to make everything as simple as possible,

and everybody always has their phone, so I can do it on a phone.

That's my goal.

I'm just saying chunky pour doughy,

and then they just go through and rate each one.

So chunky poor for treatment.

One I'll say, don't know that that one was a six

and we're just asking people for the first impression.

There's no right or wrong answers.

Usually people's first impression is right.

So that's why I'm asking people not to think too hard on it.

Maybe number two is a ten, and number three was a three.

I don't know.

They would go through all of those.

Then we would get our data and then using JMP

we would average all those ratings

and then we put the data into the data table to build the model.

So we're going to get out of PowerPoint for a second and we'll go

to excel for a second.

This is what I get when I want to export the data from Microsoft forms.

Like I said, you don't have to use this, use whatever works for you.

A s you can see, ID is the the rater number

just a random number,

not random, but just an identifier for each person.

I left it anonymous so we don't.

We don't criticize people who maybe didn't do as well as everybody else.

And in this case, this actually this data is real from.

I took this to a college class food science class and had them do this.

And so this is actual real college students rating.

The rating the the videos.

And as you can see, we have all these columns, a column for each one.

So person one rated,

rated treatment one and eight, they rated treatment,

two of four treatment three and nine, etcetera, etcetera, etcetera.

So we want to get to put this into jump.

So we have I like to use the jump add in.

So in Excel right here.

And then just as long as you're only highlighting one cell.

And you click data table, it'll import everything.

I've noticed that sometimes

I'll accidentally have like just a portion of the data

highlighted and if you could data table now

it's only going to import what you highlight.

So either highlight everything or only highlight one.

Once you hit that data table button,

you will get something like this.

So this is our data.

We need to in the end, just to show you where we're trying to get

to with this data table because we have to manipulate it a little bit.

This is our data table for the DoE.

We run it was

how much of ingredient A, B and C were in there.

I put, we'll talk about this in a minute

but I put my scale whether or not I thought something passed

or whether or not I thought something failed.

In the end, we need one more column that says Chunky pour.

We'll call it continuous.

And we'll have an average rating for for run one.

Average rating for one, two, three, four, five, etc.

If we look at this data table as it is today

is not in that format because we need all these

columns to be rows and we need the the rows to be in one column.

There's probably a thousand different ways we could do this in JMP

and they're all good and they're all correct.

I'm going to show you one way to do it.

It's just the one that works for me.

First, what we're going to do is we're going to stack

all of the columns on top of each other.

Then we're going to do a summary table

that has the average and maybe we'll also add in the standard deviation for fun.

But the very first thing that I've always been taught to do is when you get data,

you want to look at the graph, the data and look at the plot.

So we're going to actually look at the distribution really quickly.

So if we go to analyze.

There we go. Analyze distribution.

We want to look at the distribution for all of the treatments.

I'm just going to highlight them.

Go to the columns and say, okay.

I'm just looking to see is there anything

weird about this data that we should be concerned about?

When I look at so we can see for 1, 2, 3 , etcetera,

I'm looking for outliers,

like for example, three, everybody rated this sample between 1 and 6.

There was someone up here who rated it really high,

and there's also someone up here that rated this one high.

So what I like to do is if you click on this,

it'll highlight where...

So this this row represents one raider, one person.

So I'm going to see how they rated everything

and you can see they tend to be an outlier.

The nice thing is in JMP is that once you highlight one row,

all it will highlight for all the other responses.

So I can see that, yeah, they rated 3 being higher 4 being higher.

We go down, look.

Terminate. They're opposite of everybody.

It seems like for some reason

the the training, they got a little confused

and they thought higher number meant lower chunkiness and vice versa.

So what I'm going to do is since I have this row highlighted,

I'm going to close this, it'll stay highlighted.

So this is row one.

I'm just going to delete this data

and then we'll move on.

Now we feel pretty comfortable with the data is pretty much solid.

Like I said, we're going to stack the columns.

If we go to tables stack.

It's going to pop up and we just want to stack

all 17 of the treatments.

The nice thing is in JMP 17, now you get this preview.

I love the preview so then I know if I'm doing things right.

What we see here is,

as I can see,

it'll have the ID so the rater and then rate the chunky pour for treatment 1.

They gave it a five and they did number two, a seven.

This is how we want the data structured and we can change the column names.

So instead of data, we're just going to say chunky pour,

continuous.

Then for label, I'm just going to call it run because that's really

what we're going to use this for in a minute.

I just stack it.

So I say, okay, that's how I want it.

Now we have the data table in this way so now it lets us use a summary table.

S ummary tables are nice ways to be able to

make a table of the of different statistics.

So what we're going to do is we're going to highlight

the chunky pour continuous column and say statistics.

Do mean.

For fun in case we want to use it, we'll also say standard deviation.

This just gives us the overall mean and standard deviation.

But if we want to do it per run,

I'll highlight, run and put it here in group.

Now when we look at this preview, we have one through 17

and conveniently, they're in order.

One, two, three, four, five, six, seven, eight.

All the way to 17.

We have the mean and the standard deviation.

So we're going to say, okay.

Okay, so we have one more table.

Now we're to the point where we're where we need to be

because I have each run as a row

and have a column for the average column for the standard deviation.

So what I'm going to do is I will highlight this column.

If you go to edit copy with column names

and then I'm going to go to our original data table.

We're gonna make a new column here

and say edit paste with column names.

There it is.

I should have done both of those at the same time, but I didn't.

So we're going to do.

Do this one as well.

Okay, so now we are ready to do our modeling.

So first, first thing I want to show you

is what we would get if we just did pass fail our binary response.

What we'll do is if we go to analyze fit model.

Because I made this this design in JMP in the custom design platform,

it automatically knows what kind of design this is

so that's why my model is already built.

If there is a really convenient way,

if you knew this was a response surface design,

let's say, let's say this wasn't here.

The macros are convenient.

If I highlighted ingredient A, B and C.

Said Macros Response Service.

It pulls it all up. It already knows what I'm looking for.

So that's helpful.

I put it in the y axis, the variable, the response y chunky pour pass fail.

What it gives us is nominal logistic.

I'm not statistician,

so I'm not going to go into any of the statistics behind what it's doing.

I'm just going to show you how what you get out of it

and what a scientist might be looking at.

So if I say run, our target level is passed.

So when it's going to do probabilities and probability of passing.

So we say run.

This is what we get.

So, I mean, the first thing that a scientists like

myself would probably look at is this effects summary.

I'm looking at probably looking at P values and I say,

well, nothing significant except ingredient A.

There are other things that we would look at, but I'm going to...

I'm going to go over that.

We're not going to cover that today.

Instead, I want to just look at the profiler,

because that's what we find, at least in our in our experience,

the profiler being the most useful and easiest to interpret

for the scientists and when they're communicating with others.

So what this is, is I'm going to make it a little bigger.

Is on the left here.

We're going to get a probability of failing and a probability of passing.

So if we have 0.13 of ingredient, a 0.12 of ingridient B,

0.45 and of ingredient C,

and it's actually 0.13%, 0.12%, 0.45%.

I just didn't change it.

It's a very, very small proportion of the formula that we're changing

anyways at those levels,

this says 100% of the time we're going to pass.

If I move it up, let's say to..

Have like,

say point two of this ingredient now.

Now, looks like we're going to pass only 64% of the time.

You can see these curves,

how I changed ingredient B a little bit and ingredient C,

maybe we can get back up to a point where we pass 98% of the time.

You can play around with this.

But the problem with this is, is like I said earlier,

passing.

Maybe this pass right here is not the same as passing over here.

However, we don't really know that with this information,

and it's kind of hard thing for some people to wrap their head around,

like it was just probability of passing.

What do I do if if all I can get is an 85% pass rate?

Like, let's say hypothetically, this was the best we could do.

What do I do with that?

So that's why we're looking at continuous responses.

I'm just going to close this and we're going to do that,

build that model again, except let's do it for the mean

of our continuous scale.

So we're going to have to remove chunky pour

and we're going to add the average here.

We're just going to say run. Keep it simple.

Do the effects screening report.

Now you can see there's a lot more information going on

that we didn't get before.

So where before, if you remember,

all we saw was that ingredient A had a really low P value.

Everything else was like 0.99.

The conclusion was ingredient A does everything.

Well, it's not actually the whole truth, as we can see here.

Yes, ingredient is the most and most important.

The main effect of ingredient right here.

But B and C also have a role to play.

While not as big, they're still an important role.

So we look at our actual predicted plot.

It looks pretty healthy.

Our lack of fit. Look s good.

I'm not going to go into all the details of everything that we look at,

mainly because I'm not statistician.

That's just what I look at. I'll look at the lack of fit.

I'll look at the residuals to see if

there's anything weird, the studentized residuals.

Then really, I come to the profiler

and now you can see this gives us a much different picture,

much more complete picture, where as I increase ingredient A,

the chunky pore increases, but increasing these these ones does too.

So they they also have a role to play.

If we were to

say that we want to minimize it, I think it's pretty obvious what the...

Desirability is going to come out to being.

But just to show you,

we're going to you go to the red Triangle by the prediction profiler.

Optimization desirability and we're going to do the desirability function.

Then here, this is the desirability.

I find it useful.

You can change it in the red triangle,

but I find it easier if you just hit control and then click on it.

Now we can change what our goal is.

So in this case, we want to minimize this because we don't want it right?

We don't like chunky pour .

Consumers don't like it either.

So we're just going to say minimize and okay.

Now we can go back to that optimization and desirability

and say maximize it.

What I thought I was going to do.

Say, take these two ingredients out.

Put this one as low as you can.

You'll get the the lowest chunky pour that you can.

In reality, we had some other constraints, so we couldn't do that.

There were other factors at play, but this definitely gave us

a really good idea of where we needed to go,

what was important and how do we control this chunky pour

to the point where when we implemented the changes, the complaints went away.

It's been good ever since.

That is the the nutshell

of how you could take something that is hard to measure.

It's really subjective.

It's binary so you pass fail or good or bad,

and you can convert it into something that's continuous.

It's a relatively simple method.

You can use it for a number of things.

As long as you have people available to help you out,

you can you can measure a lot of things that could be considered hard to measure.

Where do we go from here?

At Hood.

Just to give you an example of some other things that we encountered.

This one, the Chunky Pour, is actually one that's a little easier to do.

But let's say this is another product we were working on a long time ago where

let's say you have coffee and you're going to add some foam to it

and you want to understand how well does that foam dissipate into the coffee?

That's a that's a tough thing to measure.

We definitely don't have any instrumentation

that can really measure it.

Videos really helped us to understand how we could measure it

and get some useful information out of...

As you can see, we're trying to measure how does that look?

How well does it move that one versus, let's say, this treatment over here?

You can see they're quite different.

Where one moves really fast, the other moves really slow.

This one looks kind of chunky the other one didn't so much.

That's that's how we use it. We use it quite often.

I appreciate you taking the time to listen to my talk.

Hopefully, I hope that this has been useful.

You'll be able to find a way that you can implement it to

in in your day to day work.

Thank you.