nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

How to see Fit Definitive Screening Design active effect decisions by stage?

CanonicalHazard

Using JMP 17.0, is there any way to see how it picked the effects it picked as active in the Fit Definitive Screening Report?

For example, the Help file says in Stage 1 with fake factor or center point replicates, an estimator of error variance independent of the model is constructed. Main effects are tested against this estimate and those with p-values less than a threshold p-value are considered active. The threshold value, unless set manually, varies by degrees of freedom. Where is this process shown?

Is that estimate the reported RMSE and the degrees of freedom below it? If so, then where are the p-values for each main effect so I can see what JMP estimates them as and how they are less than the threshold p-value for each main effect it picked? I don't see them. Instead I get a table of t Ratios and Prob > |t| and that table shows active effects for prob > |t| much larger than the p-value of 0.05 -- like a probability of 0.18, that suggests to me this Prob > |t| is not the p-value it compares to the threshold to determine which effects are active.

I'd even like to see what it estimated for all that it didn't pick as active -- didn't meet the criteria. By how much?

And the same question for Stage 2.

Sorry, my knowledge and skill involves reading a standard ANOVA table from factorial or central composite designs. I got JMP because of the special method it uses to analyze DSDs, but I am having trouble learning how to read and understand the results it gives. Stepwise analysis of my DSD gives a very different answer, picks many more effects with "Prob>F" less than 0.05., using AICc or BIC. I'm trying to reconcile the two and most importantly, understand the results for the Fit Definitive Screening Design report (prior to run or make model) that is so new to me.

Accepted answers

All comments

MRB3855

Not certain I understand your question, but in the help (as you say) it says further:

"Using YME, main effects are tested against this estimate. Main effects with p-values less than a threshold p-value are considered active. The threshold values are the following:

–For one error degree of freedom, the threshold value is 0.20.

–For two error degrees of freedom, the threshold value is 0.10.

–For more than two error degrees of freedom, the threshold value is 0.05."

In your case, DF=5 so the threshold is 0.05. That is why factor C is indicated as "active" (its p-value, Prob >|t|, is less than 0.05).

CanonicalHazard

There are 6 factors in the DSD. The table shows three of them, highlighting one of the three, and not showing the other three at all. Why those three? And where are the other three? And why are two of them listed with Prob >|t| well above 0.05? And the three missing are not shown?

I thought it was only showing those where the p value for the effect was less than the threshold p-value. If Prob >|t| is the p value, then it is showing two active effects that do not meet that threshold. And it doesn't show the other three at all. This suggests to me Prob >|t| is NOT the p value for the effect.

In that case, where is the p value for the effect that it is using to determine if that effect is less than the threshold p-value? That's my question. How do I see those p values for each effect so I can inspect that it only picked those less than the threshold p-value. And I'd like to see the other p values so I can see how far they were from being picked as active.

MRB3855

Hmmm...if you have 6 main effects (A, B, C, D, E, F, rather than three main effects and three 2nd order terms: A, B, C, AB, AC, BC or A, B, C, A^2, B^2, C^2) then I'm not sure what is going on.

CanonicalHazard

Yes, sorry, I should have said in my original question it is a 6 factor 17 run DSD depicted in the picture, but my specific DSD is not pertinent to my question.

Mark_Bailey

You might be interested in the original paper describing the method implemented in the Fit Definitive Screening procedure in JMP.

Bradley Jones & Christopher J. Nachtsheim (2016): Effective Design-Based Model Selection for Definitive Screening Designs, Technometrics, DOI: 10.1080/00401706.2016.1234979

Second, other model selection methods might be more satisfying because they act less like a 'black box.'

Third, your last example has no random error in it. Most of the design methods in JMP try to achieve an optimal design that minimizes a standard error associated with a model. The finite precision numerical methods used to estimate model parameters and standard errors often fail when the sum of squares error is zero. (It's not math, it's computing.) You should not have any such error. A computer simulation like yours is usually approached with a space-filling design and then modeled with a Gaussian Process for rapid interpolation. The idea of screening or determining statistical significance is confusing when you can look at the simulation code or have no stochastic element.