Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?I would like to evaluate the influence of 8 factors on the response. There are 1 continuous factor, 7 other categorical factors, and 1 categorical response, and 2 of the categorical factors have 4 levels, one of the categorical factors has 8 levels. Can we use the classic screening design method to design experiments?
We want to know the most influential factors among the 8 factors, where we could analyze the screening experimental results? When we add more than 6 factors or the levels of factors exceed 3, the "Screening" tab disappeared where we cannot analyze the screening design results. What are the problems? Does that mean the classical screening design cannot be used in this case? Or we should analyze the screening design results separately?
Thanks.
DOERe: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?Once you find yourself in a situation where the levels of factors is anything other than 2, the classic screening designs, that is, two level fractional factorial designs are not applicable or even doable. In your case with some factors having levels other than 2 you are forced into using an optimal DOE scenario...which in JMP takes you straight to the Custom Design platform. Since you seem to want to run a screening design I would use the Custom Design platform to build a design that is supported by a main effects only model. With an eye towards minimizing the number of runs. You may want to add a few runs over and above the JMP specified minimum to allow for some degrees of freedom for error.
Good luck trying to find active effects with a categorical response...my past experience is sometimes you need lots of runs to tease a signal from the noise...but you'll never know unless you try.
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?Thank you very much. And I have another follow-up question.
I used the Custom Design platform. Since I have a binary response, when I fit the model, I chose Personality: Generalized Regression, Distribution: Binomial, Estimation Method: Elastic Net -- Adaptive, Validation Method: AICc. But when I click on the "Go" button, it shows an alert which is "Fitting terminated early because of failure to maintain heredity. Consider running again without enforcing heredity". Is this a severe problem I have to solve, how could I solve the problem, or should I choose some other methods?Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?This refers to the fact that the model hierarchy is important. JMP strongly encourages you to maintain the hierarchy, but does not require it.
Model hierarchy means that if you have X1*X2 in the model, then X1 and X2 should also be in the model, even if they are not significant alone.
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?@yiyichu Great that you used the Custom Design platform. Can you share the specific model you articulated for the design construction? Was it main effects only? Or something different?
In my first reply, I suggested you specify a main effects only model...the error message you are seeing and based on @markbailey 's correct reply, it sure looks like you are trying to fit a model with more than just main effects? Does your design support at least some level of estimation for at least some two (or heaven forbid) even higher order factor interactions? With categorical factors at more than 2 levels, to estimate interaction effects containing those factors your design can get big in a hurry. Especially with the one factor with 8 levels.
And I'm also just a bit curious regarding why you chose the Elastic Net fitting personality? I guess, in a screening modeling mode, I might have started out with perhaps just a simple nominal logistic regression model. Elastic Net isn't an incorrect choice...but it might tend to zero out some of the parameter estimates since that's it's very nature and you might miss effect level information content. Just curious more than anything?
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?I am also curious, but I'm curious why you are looking at 4 level and 8 level categorical factors at all. This sounds more like a pick the winner test than a screening design? My thoughts: Screening is not meant to test all possible combinations. Can you pick the extremes of the 4 or 8 levels? The idea would be to compare all of the factors (equally, without bias) and determine which of those are worthy of further investigation. By testing at more than 2 levels for some of your factors, you have more information about those factors and have therefore biased your study.
But, of course, I don't know your specific situation, so you may want to ignore me.
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?@P_Bartell I think it should be the main effect only previously, but I may mess something up. I recheck the model and make sure the model to be the main effect only, and it seems the error goes away now.
Since I am a starter of DOE and JMP, I chose the Elastic Net fitting personality just because I follow the example that was provided in the JMP Documentation. I will try to use the basic Logistic Regression estimation method as a starting point then. Thank you for your advice. When we chose the estimation methods, how do we decide what estimation methods we should choose, when to choose a basic logistic regression estimation method, when we should choose others, like Lasso, Elastic Net, Ridge, etc.
And from your first reply, you said sometimes we need lots of runs to tease a signal from the noise for a categorical response... Could you please explain a little bit more about that? What could we do to check whether we need more runs?
Thank you for your time again.
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?@statman The 4-level factor here is "motion level", including Major motion, Minor motion, Fine motion, and none motion. The 8-level factor here is to randomly choose 8 different locations in a room for our case. But you are right. Actually, I am also considering picking the extremes of the 4 level and 8 level factors to make all the factors at 2 levels. Then we may also choose the definitive screening design method. I am just not sure which way could be better. Now your advice makes me more confident to do it this way. Thank you.
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?For Motion, pick extremes. If that is significant, then you can "fine tune". For location in room...Can you actually control this or is it noise? If noise, consider blocking.
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?@statman Thank you for your advice.
If I chose to use Method 3: RCBD as you mentioned to replicate the treatments over 2 blocks (best and worst locations), it is like the following:
The default number of runs is 16, and then the "Color Map on Correlations" is as follows:
From the above color map, it seems the model is not the main effect model. In general terms, the color map for a good design shows a lot of white off the diagonal, indicating orthogonality or small correlations between distinct terms. Does this mean that this is not a good design? And if I increase the number of runs, it will show most of white off the diagonal, whether it means I need to increase the number of runs when I do experimental design?
And one more question here:
Our response is a binary value (0/1), 0 indicates non-presence of occupants, 1 indicates presence of occupants, for the response, I used "Maximum" as the Goal, as follows:
Is this proper, or should I use "None" as the Goal?
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?@statman Thank you for the suggestions.
I tried to find these plots, including Daniel plots (normal plots), Pareto plots of effects and possibly Bayes plots, but it seems they are under the "Effect Screening", and only available if you use "Standard Least Squares" as the "Personality", which means the response should be continuous. There is no such choice if I chose the "Generalized Linear Model" with the "Binomial" response. Am I wrong, or do I need to find this somewhere else?
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?While both @P_Bartell and I are trying to give you some advice, you should realize there is no "right" design. Every design will potentially vary in in the resources required, the precision of detecting design factor effects, the resolution, the model that can be estimated, the number of factors that can be estimated, etc. What I recommend is to design multiple experiments and evaluate them against the set of criteria and your situation. I also suggest you predict all possible outcomes and what you will do in each outcome. You have 8 design factors and at least 1 noise variable. The purpose of this 1st experiment is to design a better experiment (move the design space and identify the factors that are most interesting). Usually a lower resolution will work to identify main effects, but you should rank order the model effects up to 2nd order to determine whether you need to bump the resolution. Also when you have JMP block the design, the software treats the block as a random effect by default. However, when you know what noise is making up the block, you can treat it as a fixed effect and quantify block-by-factor interactions. I'm a bit old school you might say. I want to know the aliasing and don't like to do partial aliasing as this is more difficult to interpret and determine what the subsequent experiment should be. But I am probably in the minority here. Many folks like the optimal designs for their efficiency. I also don't fill in any Y's as I don't want the software to "control" my analysis (in some cases, limit my analysis options).
Regarding your response variable, have you thought about other ways to quantify the response? The binary response requires large sample sizes to detect differences. Could you vary the size of the "object" or "person" and record what size object gets detected? Or could you vary the amount of "motion" it takes to get detected? If distance from sensor matters, instead of blocking on location in the room, could you determine what distance from the sensor the object is detected? In any case, I would recommend more thought on response variables.Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?Yes, you are correct...sorry for missing that the response was binary. I really suggest you create a better response variable (see previous note).Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?Sorry to join this discussion so late.
The Location seems to be a categorical factor with fixed effects. Is it really a restriction on availability? That is, you can only make a limited number of observations per location? How many locations are at your disposal? The fact that you called the two locations "best" and "worst" suggests a qualitative or categorical factor to me.
The model by default is only the main or additive effects. The color map shows you higher order terms, too, so that you can assess the impact of these effects (correlation) on the main effects. The effects will bias your model if you do not include them and increase the standard errors of the estimates if they are included.
What is the response? Is it binary or is it a count of two binary outcomes? For example, I might observe 20 instances in each run and record the number of each outcome (succeed, fail). On the other hand, I might only observe 1 instance in each run. You cannot use the power analysis in custom design in the case of such a response. But you certainly need more runs for such a response. The Simulate Response command in Custom Design along with the Simulate feature in JMP Pro can be used to assess the empirical power of any design.
The goal for the response is saved as the Response Limits column property. It defines the desirability function for optimization in the prediction profiler. I do not think it is appropriate for this case, but you can change it after you get to the profiler, and then save the updated definition.
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?@statman Based on your idea here, the block design will bring partial aliasing since it will quantify block-by-factor interactions. And we don't want to do the partial aliasing for the first experiment, am I right? And you said, "you didn't fill any Ys", does it mean you didn't input "response"?
Our response variable is binary Presence/Non-presence, which means whether the occupancy sensor detects people or not in space. Actually, the other way we could interpret the response is to count the number of people in a testing room. And then we could decide the sensor Presence/Non-presence based on the results later. If the count number is 0, then "Non-presence"; if it is not 0, then "Presence". Other than that, I couldn't think of another way to interpret the response. Or maybe you have some good thoughts? Would this be helpful to improve the case? Unfortunately, we couldn't detect the size of objects/people by using the occupancy sensor. The motion has four categories, major -- people walking, minor -- people extending their arms, fine -- people typing, none -- sleeping. What do you mean by varying the amount of motion? And the location influences the performance of the sensor not only because of the distance but also the angle from the sensor to people. If we want something to replace location, maybe we should include both distance and angle variables. And then we define the extreme values for distance and angle at two levels. Would this be helpful?
Re: Could Screening design used for 1 continuous factor, 7 other categorical factors, and 1 categorical response?@markbailey Thank you for providing your thoughts.
For the location, the background is we set up the occupancy sensors, people will stand in different locations in a testing space, and then we evaluate whether the occupancy sensor will detect occupants or not. Because we want to know whether people's location would impact the performance of the occupancy sensors. Initially, we chose 8 random locations as 8 levels for the location variable. But base on our discussion with @statman, it is better not to use 8 levels for the first experimental design, that's why we decided to choose the extreme cases for the location, one is the worst case where the sensor couldn't detect occupants, one is the best case where people just stand right before the sensor so the sensor will detect occupants perfectly. So maybe it is not a restriction on availability? It is just a variable we want to vary to see how it impacts the response?
For the response, it is a binary value that we observe 1 instance in each run. For our case, the response is the Presence/Non-presence of people. Presence means the sensor detects occupants, Non-presence means the sensor doesn't detect occupants. So in this case, what should I choose for the goal of the response?
