nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

DoE & D-Optimal design

flo_kussener

Hello everybody,

I have a lot of question from a customer who is evaluating JMP for DoE. He compares it to Nemrod and he would like to check different points. I'm not confident about a set of questions which are just below.

If anybody had an advice, it would be great.

1. Would it be possible to find the best number of trial for a D-Optimal Design?

I mean, when you do a D-Optimal, you choose a number of experiment, and JMP Compute the best points for this design. You can ask to JMP to do the same with another number of experiment and compare the quality of the two design, and you can check if adding some trials is relevant or not.

The question is how to find the best number automatically. It means saying the minimal and the maximal number of experiments that we accept to do and find the best number in this range. Furthermore, as it is a matrix optimization which compute the point, the iterative methods can do several time the test for the same number of experiment.

Surely, we can do it by scripting. Does this kind of script is already existing? Is there another way?

2. Is it possible to say to JMP that we want to do absolutely a predefined experiment and to add additional point. For me the Augmented Design Option is the best approach, but perhaps there is another way?

3. More Generally, and in this case, the Augmented Design Option is not useful: Is it possible to find a D-Optimal Design on a set of Experiments already Defined. So the ability to pick up a subset of a table optimally.... We can apply the same question for an uniform Design. I would think to constraints but it seems that it is not enough and too complex when we have hundreds of available points.

4. For the analysis, the suggested model is related of the definition of the Design (I mean, only main effects, etc). We can change that and it is good in certain cases (for example DSD), but it is possible to block things to be sure that people are not evaluated bad models by adding too much things?

Thank a lot for your suggestion,

Florence

Accepted answers

All comments

Phil_Kay

Hi Florence,

There are a lot questions here. I will try my best to answer.

1. I don't fully understand this. I think they are asking to find the best experiment between the range of say 8 and 16 runs. Perhaps they think that there is an optimal number of runs in this range. However, it seems to me that more runs will always be better (depending on how you define "better"). The optimal number will always be the highest number in the range as more runs = more data = more information.

2. Yes the Augment Design option is the approach to use here. Is there a problem with that approach?

3. In this case they need to define the factors in their table as covariate factors in Custom Design.

4. No. I'm not really sure how you would predefine what you would want to block. How do you know what is a "bad" model before you have the data. I am guessing that what they mean is that they would like to stop the user from being able to add cubic terms or 3-factor interactions to a model, for example. You could talk about how customers like Unilever have built applications to guide less experienced users through a particular workflow for DoE.

Phil

louv

Hi Florence,

Agree with Phil's response. Additionally it appears that in the first question there is a buried question of signal/noise of the system. The number of tests or treatment combinations in the design depend on the signal/noise of the system being examined which includes the total noise of the process + experimenter + test. He may need to be made aware of the number of starts capability for the algorithm but as Phil states more experiments = more information. The real question is how many experiments can one afford.

The augment experiment is one way but if one wants to test a specific setting then clearly just a manual addition of the row with the settings is the way to do this.

The protection against the wrong model is verification runs to test the model. If there are too many terms or erroneous conclusions than that most certainly would be seen when the verification run does not perform as expected.

Lou

Peter_Bartell

Florence: I too agree with everything Phil and Lou have contributed. One other thing I'll add is have the customer take a look at the Design Evaluation options...lots of different ways to compare one design to another in there from a power, color map on correlations, and other perspectives. Picking a 'best' design is almost always a mix of multiple either statistical (power) or practical (budget) criteria.

malcolm_moore1

Thanks for your comments everyone. Florence is out today and I just wanted to update this thread with our discussion from yesterday in case there are other suggestions for the outstanding issues:

1) We think he is after a slightly different optimal design algorithm – one that allows user to specify model, total noise variability, alpha level and power and system returns to the user the required sample size and resultant DOE worksheet. Apparently Nemrod offers this. Might be nice to add this?

3) This scenario is to select a subset of design points from an existing large design or set of runs. I thought we used to provide ability to provide a candidate set of runs for Custom Design to select from, but I can’t find it anymore – maybe it was RS/Discover! Looks like our option is to use observation clustering on bigger design set to define a number of clusters equal to preferred smaller design size, then select one row from each cluster for the actual DoE. Other suggestions appreciated. It would be nice to have the ability to select a design from a predefined set of candidate points.

4) The goal is to prevent users from choosing a model type (Interactions, RSM) that results in a very large number of runs when there are a large number of factors when using Custom Design. One solution is to use Screening Design instead and select the second radio button under Choose Screening Type “Generate orthogonal or near orthogonal designs for main effects only”, this then forces same design as custom design with a linear model. This feature is limited to linear models only, so other suggestions welcome. Might be nice to have option of disabling interaction and RSM model effects when number of factors is large. Could even be a preference setting for threshold on number of factors to enable this setting.

Bill_Worley

Just some thoughts on number 3.

In the past I have used either a 3D scatterplot or a scatterplot matrix of the design points to visualize the design in the various dimensions. You could then use the selection box or point selection tool to subset the design.

Another option is to use the row selection tool and choose "Select Randomly" or "Select Where.." if you want to be very specific in the range(s) for the inputs.

Best,

Bill

flo_kussener

Hey,

I want to thank you about all your comments.and thank Malcolm to update my question.

Here an additional input:

1. Yes, for Custom DoE each experiment add information. The fact is to do as in a stepwise with p-value: at a level number of row, an additional row will add less information than before. His need is to find this level. The impact of new row is not linear on the power analysis. So the question is to find the inflection point. Clearly, it is scriptable.

Thank again,

Florence

De : JMP User Community Administrator <jmpcommadmin@sas.com<mailto:jmpcommadmin@sas.com>>

Répondre à : "jive-882643824-hwqt-2-4nv6@community.jmp.com<mailto:jive-882643824-hwqt-2-4nv6@community.jmp.com>" <jive-882643824-hwqt-2-4nv6@community.jmp.com<mailto:jive-882643824-hwqt-2-4nv6@community.jmp.com>>

Date : vendredi 24 juillet 2015 17:04

À : Florence Kussener <Florence.Kussener@jmp.com<mailto:Florence.Kussener@jmp.com>>

Objet : Re: - DoE & D-Optimal design

JMP User Community<https://community.jmp.com/?utm_source=JIVE&utm_medium=email&utm_campaign=System Email Tracking>

DoE & D-Optimal design

reply from billw@jmp<https://community.jmp.com/people/billw@jmp?et=watches.email.thread> in JMP SE Community - View the full discussion<https://community.jmp.com/message/217554?et=watches.email.thread#217554>

malcolm_moore1

Big thanks to Marie Gaudard for solution to item 3. It is to use the covariates option when defining factors in custom design, e.g. take an existing table with a large number of factor combinations, select all factors as covariates within custom design, specify your model, then under design selection specify a smaller number of runs (default is number of rows in table). Result is a design that is a subset of the rows in the table. See last script in diabetes table attached.

Malcolm

Diabetes.jmp

Phil_Kay

Regarding #4: I have come across a few examples where customers have got in to difficulty because they are trying to create a design with lots of factors and are specifying a high order model, especially where one or more factors is multi-level categorical and they are asking for all interactions. The computation becomes veeeerrrryyyyy slow. Perhaps a warning could pop up to tell you that you have a high number of effects in your model (similar to all possible models in stepwise). There could also be the choice to remove interactions and/or higher order polynomials.

Regarding#1: That does sound scriptable. I guess they could look at how average power improves vs number of runs. The challenge for a more general solution would be that there are so many different ways to measure the "goodness" of your design. I suppose it would be possible to create a dashboard that would enable the user to look at how different measures change with number of runs.

flo_kussener

Hey,

I did a (very simple) script for the first point. So they can see the workflow…

///////////////////////////////////

power = J( 10, 1, 0 );

x = J( 10, 1, 0 );

For( i = 11, i < 21, i++,

design = DOE(

Custom Design,

{Add Response( Maximize, "Y", ., ., . ), Add Factor( Continuous, -1, 1, "X1", 0 ),

Add Factor( Continuous, -1, 1, "X2", 0 ), Add Factor( Continuous, -1, 1, "X3", 0 ),

Set Random Seed( 25644839 ), Number of Starts( 22106 ), Add Term( {1, 0} ), Add Term( {1, 1} ),

Add Term( {2, 1} ), Add Term( {3, 1} ), Add Term( {1, 2} ), Add Term( {1, 1}, {2, 1} ),

Add Term( {2, 2} ), Add Term( {1, 1}, {3, 1} ), Add Term( {2, 1}, {3, 1} ), Add Term( {3, 2} ),

Set Sample Size( i ), Optimality Criterion( 2 ), Make Design}

);

r = design << report;

r2 = r[Outline Box( "Design Evaluation" )];

// comparaison sur le premier effet principal

power[i - 10] = r2[Outline Box( "Power Analysis" ), Number Col Box( 1 )][2];

x[i - 10] =i;

Show( power );

);

New Window( "Power Evaluation",

Graph Box(

FrameSize( 200, 200 ),

X Scale( 10, 20 ),

Y Scale(0, 1 ),

For( i = 1, i < 11, i++,

Marker( 1, {x, power} )

);

)

);

//////////////////////

Thank again for all your suggestion!!!

Florence

Ryan_Lekivetz

Good suggestions so far.

For #1, you're above script is the extent to what we've done, but as above, generally the more runs the better, it just becomes a question of if the additional cost is worth the added benefit.

We might be able to make this more slick in JMP 13 with Compare Designs, and we've been discussing ways we could get it to work by varying run sizes.

For #3, you can sometimes force in a subset of design points by creating a new categorical variable, say "defined" with different levels for the points you want in vs. those you don't (maybe "yes"/"no"?). The ideal is that the number of defined points is less than the number of additional points you want, so all the "yes" will be forced in. Otherwise you may need to break the "yes" up into "yes1", "yes2" to get them all in the design. Just make sure the "defined" is one of the model terms.

I've seen this question come up from a few customers, where they have specific candidates they want in the design, so in JMP 13 we'll have an "enforce use of selected covariate rows" option. It's just in custom right now, but hopefully we can get it into space-filling as well.

Cheers,

Ryan

Phil_Kay

Thanks, Ryan.

That is all good to know.

Phil