nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Repeated measurements...sort of...

gchesterton

Suppose I have a 2^3 full factorial design with replicate, so 2x8=16 runs. Each run lasts 30 minutes, where participants are detecting and prosecuting 8 'targets' over the course of that 30 minutes. Each of the 8 targets can conceivably have its own response time. Since the target profiles vary, I am randomizing their order for each run. I don't care about a target type factor, so it is not a design factor for which I am trying to measure an effect. I am trying to detect main and 2-way effects.

Does it make sense to treat those 8 response times per run as independent observations, as if I was getting 8 response time data points for each factor level, for a total of 16x8 observations? Or instead average the 8 response times for a single observation per run? If I can use 8 observations per run, are they best treated as repeat measurements for a run? Because they're not repeated measurements -- each target is unique and I know in advance they will not produce the same response time. All 8 are subject to the same treatment of course, but they vary in other ways and it is a source of variance.

Accepted answers

Mark_Bailey

Yes, enter the response for each target as a separate observation (row). Add a Profile column with the nominal modeling type. Add the Random Effect to this term in the Effect list. Do not add any term for the replicate - that will be used for the error estimate (repeatability).

How does that work?

All comments

Byron_JMP

Trying to understand your experiment.

It sounds like you have 16 experimental units.

Within each unit some action is repeated 8 times.

1. All 8 actions are dependent on the conditions of the experimental unit.

The goal of the experiment is to measure the effect of the experimental conditions on the measured actions.

or...

2. The experiment results in a response curve with 8 points over 30 min.

The goal is to understand how the experimental conditions affect the response curve.

I think you might be working with case 1. Stacking all the rows for the 16x8 data points is fraught with danger.

I suspect you have your data table set up one column for each experimental variable, and one column for each of the 8 measurements. One option is to treat this as a mixed model and analyze the between and within variation in the 8 times. Another option is to use the mean and standard deviation of the 8 measurements as the responses.

gchesterton

It's definitely case 1 - The goal of the experiment is to measure the effect of the experimental conditions on the measured actions. I too thought it would be wrong to stack the observations as if I had 8 observations for each factor's level.

If I were to take the mean and s.d. of the 8 observations, then my response variable becomes...the mean? So would I then have 8 means for both levels of a factor, and testing for a significant difference of the means of those means?

gchesterton

I should have avoided causing confusion in the example I provided by instead supposing that I have 10 targets per run. If those were somehow expressed as a single summary value (mean), I would have 1 mean for each of the 16 runs. If I treated them as separate values, I would have 10 values for each of the 16 runs. I'm trying to think ahead for how I would be analyzing the results in both those cases if my goal is simply to verify a factor effect (for each of the 3 factors).

Byron_JMP

Taking the mean of the (10) replicates for each run (1 of 16) will give you a better estimate of the mean time to acquire a target, or time between acquisition, however you are scoring the events...

If you wanted to also account for the order of the targets, in the case of the targets not all being identical, there is a little different experimental design for that. (experimenting order factors).

At the 2019 discover conference, Kevin Gallagher, a Scientist at PPG Industries, presented a really nice case study on this topic. https://community.jmp.com/t5/Discovery-Summit-Tucson-2019/The-Design-and-Analysis-of-Experiments-With-Order-Factors-2019/ta-p/223351

Mark_Bailey

Alternatively, you might consider target profile a factor. That is, it has an effect on the response. It sounds like you do not consider target profile to have a fixed effect. While each treatment includes the same eight profiles, the profiles represent a random sample from a population of possible profiles. They introduce additional variation to the response, a random effect. This approach would give you additional information: variation across and within profiles.

Byron_JMP

@Mark_Bailey How would you structure the data to include target type as a random effect?

gchesterton

Thanks @Mark_Bailey , I think that's the right way to think of it. I am re-using the same 10 target profiles, just randomly varying their order to reduce operators' predictability. I want to be able to remove the target profile source of variance, but I don't need to measure the target profile effect. So, does that mean I simply have a profile variable (column), and produce 10 observations (rows) per run...one observation for each target profile, for each of the 16 runs? That would mean a data table of 10x16 = 160 rows if I'm following your logic. Do I need to tell JMP to treat the target profile variable any differently either in building the design or in the ANOVA?

Mark_Bailey

How does that work?

gchesterton

Thanks again @Mark_Bailey I asked JMP to create the table with some fake data, then modeled the results just to implement your recommendation. With the output table, I ran two models -- one model with the three main effects terms and three 2-way terms, and the target profile term as a random effect, and one model where I simply excluded the target profile term from the model. In the first model dialog, JMP recommended REML. I am using the SLS personality and effect leverage emphasis. In the second model, without the random effect term, JMP did not prompt me for a method.

In any case, the results in both models are quite similar -- perhaps because the data were meaningless since they were synthetic. All main effects and interactions were significant at 0.05 in both models. That seems counterintuitive -- its seems like treating the profile as a random effect term should have yielded much different results. But perhaps I am comparing apples to oranges in the way that I am modeling the data with these two models. In model 2, I am probably making bad conclusions since there's an effect I didn't account for. But I would have thought that would have resulted in fewer significant effects.

statman

Late to the party, but there are a number of things that come to mind, though I can't say I understand what exactly you want out of the experiment.

But this sounds interesting...so if you'll indulge me I have some questions (of course if you are happy with the responses you have received, I understand):

What are responses are trying to understand? Response time? Rate of response (from detecting to prosecuting)?

What is a "target profile"? And why don't you care about the interaction between the design factors and the target profile?

Aren't you trying to choose design factors, that affect response time, robust to target profile?

It seems you're only interested in the mean? Why not the variation? It is impossible for me to think about means without some idea of the variance...it is like giving the score for one team and not the other in a sporting event.

If you are using the mean, shouldn't you first test to see if that is the appropriate statistic? What if there are unusual data points in the 8 target profiles? Then the mean might be a poor summary statistic.

You are replicating the design, why? Do you know what noise is changing between replicates? If so, wouldn't you be interested in knowing the effect of that noise and possibly noise-by-factor interactions (think robust design). Treat the replicate as a block and a fixed effect. If not, then I understand the use of the replicate to get an "unbiased" estimate of the MSE.

A further thought is to treat the design structure as the whole plot and the profile factor as the subplot of a split-plot design. I would think you would want to know more about this source based on the statement. "All 8 are subject to the same treatment of course, but they vary in other ways and it is a source of variance"

gchesterton

Hi @statman jump on in.

@statman wrote:
What are responses are trying to understand? Response time? Rate of response (from detecting to prosecuting)?

It's a command & control scenario where detectionTime is one (of many) measures of the effect of several design factors.

What is a "target profile"? And why don't you care about the interaction between the design factors and the target profile?

The target profiles add variety and reasonable error to the experiment, but they are simply a small sample from a large number of potential profiles. I don't really need to know whether there is a target profile effect.

Aren't you trying to choose design factors, that affect response time, robust to target profile?

well, yes

It seems you're only interested in the mean? Why not the variation? It is impossible for me to think about means without some idea of the variance...it is like giving the score for one team and not the other in a sporting event.

Hence my earlier concern that collapsing a bunch of values to a mean does not seem like a good idea. Perhaps a 90th percentile is more approp if I am trying to characterize a sample with a single value.

You are replicating the design, why? Do you know what noise is changing between replicates?

It's just the randomness associated with human operator actions. Two runs with the same factor levels will definitely yield different results, because the response is affected by human actions and decisions.

A further thought is to treat the design structure as the whole plot and the profile factor as the subplot of a split-plot design. I would think you would want to know more about this source based on the statement. "All 8 are subject to the same treatment of course, but they vary in other ways and it is a source of variance"

I tend to think of split-plot designs for cases where I have a hard-to-change factor, which is not the case here. This is a simulation where we can control the factors fairly easily from one run to the next.

statman

You need to read "Split-plot designs for robust product experimentation" Box and Jones, Journal of Applied Statistics, Vol. 19, No.1, 1992. Hard to change factors is only one reason to do split-plots (and not the best reason). Restrictions on randomization can be used effectively to partition the noise in an experiment to simultaneously increase precision for detecting factor effects while not negatively affecting inference space.

When you are calculating the mean to use as a response variable, also calculate the standard deviation (or variance) and us eat as a response variable.

It seems you ultimately want to understand human actions and decisions. Randomizing those will not allow for estimating those effects nor will it allow understanding of those effects.

gchesterton

@statman

Ok, so in the split-plot option, I suppose I'd develop say, 6 scenarios that involve various target profiles. Those scenarios would be the main plots and would occupy a day. Then within those main plots, I would have split plots for levels of the system design under investigation: system A, system B.

day	1	2	3	4	5	6
am	A	A	B	A	B	B
pm	B	B	A	B	A	A
Scenario	5	2	3	1	4	6

What are the real benefits of this approach, if as is my case that changing the scenarios is not difficult?