Means comparisons for repeated measures on survey dataWhat is the most efficient way to compare means of repeated measures of survey data? Right now, it seems there are 2 main ways- matched pairs and fit model (with panelist ID as a random effect). However, in many of the tests we run, we have many means to compare and I am looking to see if there exists any additional functionality in the Related tab in categorical to speed up this process.
I know there was a discussion previously on this topic (titled: Questions about Multiple Response type in Categorical Platform) and I am looking to see if there are updates.
Re: Means comparisons for repeated measures on survey dataYou’re right. The Categorical platform does not currently handle all types of repeated measures. As you requested, we did implement overlap corrections for repeated measures for the Repeated type of aligned responses. I do understand that this does not cover all cases of repeated measures and that it’s desirable to handled repeated responses.
As noted, SAS can handle these appropriately. As such, one solution would be to write a JMP add-in that connects to SAS and passes the data to be analyzed and then the results presented in JMP.
I would point out, however, that these analyses are complicated. Often, there are not enough repeated measures per subject to estimate a repeated measures model. The example you posted here is a prime case.
We don’t currently have any plans to implement generalized linear mixed models in JMP, which is what PROC GLMMIXED does. We do understand your situation and desire to do this in JMP but that type of model isn’t currently on our roadmap.
Re: Means comparisons for repeated measures on survey dataContinuing the repeated measures topic; it seems that the consumer goods industry is continuing to move towards increased numbers of test designs in which repeated measures test designs are being utilized. Based upon the previous response, it seems that the current categorical approach suffers from the repeated measures assumption and the mixed models approach has issues with treating the categorical responses as continous. Can you shed any light on whether or not future versions of JMP might consider platform improvements to cover this needed approach?
Re: Means comparisons for repeated measures on survey dataScott,
The reason the output is so different is not because of the overlaps assumption but because the two are very different models. The Categorical platform is treating Looks Good as a Categorical variable. It has the continuous modeling type, but putting it into a contingency table is basically asking JMP to treat it as categorical anyway.
I can see why someone might choose to treat “Looks Good” as continuous. JMP doesn’t offer mixed models for categorical data. There also aren’t as many options for multiple comparisons in the statistical literature as there are for continuous data. If you were really worried about the repeated measures assumption in this case, you could use PROC GLIMMIX in SAS. I did try estimating a mixed model for Looks Good using PROC GLIMMIX, and the model wouldn’t converge. This isn’t surprising. You only have two observations per person. An ordinal model for this data requires 4 intercept estimates. Estimating a random intercept for each person isn’t possible for an ordinal model with this data.
I also tried a version where I re-coded Looks Good so it has 3 levels instead of 5. This model does converge, but only with warnings. It’s my opinion that a mixed model isn’t a good solution for this data. The attached PDF shows both the SAS code and the output (the community software doesn't see text files as valid uploads).
“Looks Good” only has 5 distinct values (and, really, only 3 of those are usable because hardly anyone answered using the last two levels). It should have the ordinal modeling type. If it did, Fit Model would product an Ordinal Logistic regression (table with example attached). If you do that, the pvalue for Product A vs Product B is nearly identical to the one produced by the Categorical platform. (It won’t be exactly identical because there is an extra predictor in the Fit Model version.)
In my opinion, for this example, treating a categorical column as continuous is at least as bad as violating the repeated measures assumption. The “Actual By Predicted” plots at the top of the Fit Mixed output look pretty bad. The p-values in the output are misleadingly low because the model isn’t complicated enough to fit the data, and so you the p-values in the hypothesis tests are based on assumptions that are not true. JMP is doing the “Right thing” according to what you told it to do, but it does assume that you gave it the correct instructions and that the user checks the model assumptions using the output provided.
Re: Means comparisons for repeated measures on survey dataI'm interested in knowing whether JMP is going to develop anything in this direction of handling crossover studies or other repeated-measures like studies where multiple treatments are involved. I see a variety of repeated structures here. Many of them can be handled using the Mixed personality in Fit Model but none can be handled by Categorical today...
Re: Means comparisons for repeated measures on survey dataMelinda-
Attached is a sample data file for a 2 product repeated measures test. This is a typical format we receive from our suppliers. We have an add-in that automates much of the procedure and displays the results in a manner more consistent with the Consumer Research> Categorical> Structured format. I would like to share that with you to show you how it works and what it is doing, but would prefer a direct sharing vs. community sharing if that is possible. Also for perspective, I removed the majority of the columns/ questions but that is the reason for asking for a way to do more than one question at a time.
Thanks,
Scott
Re: Means comparisons for repeated measures on survey dataMelinda-
Thank you for replying. I am looking for a data set to try this out and see what this could look like. Separately, we have some add-ins that get us close, and I can demonstrate how we use those if that helps to illustrate my example.
Regarding the overlap, I will also pull a sample data set to show you a potential issue to see if you can advise on I can use it properly.
Thank you,
Scott
Re: Means comparisons for repeated measures on survey dataHi, Scott. I have seen a few requests for an overlap correction for means comparisons for check all that apply.
We added an overlap comparison to the Repeated measures for means comparisons in JMP 13.0 at P&G's request. That feature only works if you don't have an X column specified.
You say you have "A lot of means to compare"--Would it solve your problem if we added support for X columns to the overlap comparisons on Repeated?
