<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />varbanov.ar passed along some questions regarding the Multiple Response option in the Categorical platform.
I'm answering them here to provide for a wider audience.
Requests/Questions for JMP Categorical Experts:We need more clarity on the different Multiple Response options and output in JMP Categorical. Can someone of the JMP Categorical Experts give us a short demo with answering some questions about it? Is there a technical report with some more explanation for the different statistical test used by this Platform. In particular, we have also the following questions:1) What is the definition of multiple responses?
In JMP Categorical, multiple response data is derived from survey items (or other processes) in which the response categories are not mutually exclusive. A common source of multiple response data is “check all that apply” survey items. Multiple response does not refer to repeated measurements data.
There are several ways that multiple response data can be represented in a data file. The data may be long – i.e., there is a row for each unique response item and respondent id combination (see Sample Data Library > Quality Control > Failure3ID.jmp > “failure” column as an example). The data may be wide – i.e., each row represents one respondent and there are multiple columns that indicate the selected response options (see Sample Data Library > Consumer Preferences.jmp > columns “Brush After Waking Up” through “Brush Another Time”). There may also be one row per respondent and one column that includes a delimited list of the selected response options (see Sample Data Library > Consumer Preferences.jmp > column “Brush Delimited”). With the latter approach the “Multiple Response” column property can be assigned to the column and the appropriate corrections for overlapping samples will be applied to statistical comparisons when the column is used as a “top” variable in a structured table. (Note: It is easy to move between the last two formats using the “Make Indicator Columns” and “Combine columns” utilities under the Cols menu.)
More information on using multiple response variables in the Categorical platform is available in Help > Books > Consumer Research, starting on pg. 62
2) Is there a proper mean score comparison between two groups within JMP Categorical in presence of overlapping samples (e.g., each consumer evaluated both products)? If yes, what is the way to get it (Multiple Response tab does not seem to provide it)? Also can this analysis be done in the structured tab?
The Categorical platform does not correct for overlapping samples, as you've described them, for means comparisons.
What you're describing as overlapping samples is what we think of as repeated measures.
There is a “Repeated Measures” option under the “Related” tab in Categorical. In line with what one would expect from the Categorical platform, it reports things like frequencies, share charts, means and other summary statistics. However, it does not do comparison tests.
Repeated measures means comparisons are available in other areas of JMP:
- The Matched Pairs platform will compare and test two means (such as product ratings) at a time. However, in the presence of missing data it does listwise deletion, removing the entire case from all of the comparisons.
- Since you have missingness in your data (but even if you didn’t), you might consider using Fit Model with a mixed model with the product ID as a fixed effect and the respondent ID as a random effect. The example data they sent is already in the correct layout for this approach, but not for the other two approaches above.
- Obviously, the actual appropriateness of any of these methods is dependent upon many details, so these are just general suggestions– your mileage may vary.
3) What statistics is JMP using when you invoke the categorical variable as multiple response?
Remembering that the Multiple Response type is different from what you've described as "overlapping samples" here is what is done:
- When a column that has been assigned the “Multiple response” property is used as a “top” variable in a structured table, a first-order Rao-Scott adjustment is applied to account for overlap (e.g., Lavassani et al., 2009, Developments in Analysis of Multiple Response Survey Data in Categorical Data Analysis, Journal of Applied Quantitative Methods). Tests are handled differently when the table is created in the “Multiple” tab; the correction is only applied when the table is created via the Structured tab.
- When a variable with the multiple response property is assigned to the “side” dimensions there is no correction for overlaps.
- The statistical tests are as follows:
- Compare Each Cell: These are Fischer Exact tests. These tests are no different from other responses categories.
- Compare Each Sample: Is a Pearson test that uses the Rao-Scott Correction (when the multiple response column is assigned to the “top” position)
- Test Response Homogeneity: Has two options.
- Binomial: Treats each row as if it were two rows. One row for “checked that box”. One row for “did not check that box”. This option was added based on a previous P&G request.
- Poisson: Treats the counts as if they were being modeled as the dependent variable in a Poisson regression. The columns in the table are treated as if they are levels for the independent (X) variable. The p-value is equivalent to the p-value one would calculate for an overall model test calculated in this way.
4) Within the categorical platform, it is possible to provide p-values for the mean comparison?
The ability to save t-test and p-value reports was added previously based on a P&G request. It is possible via the red triangle menu.
5) Is the general formulae that “adjust” for overlapping samples implemented as used in Desktop Reporter Software? Is there an efficient way/data structure/grids to “tell” JMP that there are overlapping samples?
As noted above, using Fit Model to fit a mixed model may be the most appropriate way to handle this in JMP.
6) Jeremy’s issue: (JMP contacted on this before by Mike Creed/Diane Farris)a. When doing the calculation, it appears that JMP is not adjusting the degrees of freedom when using the Rao Scott first order correction.
We believe there's was some confusion between "overlapping samples" and "multiple responses" but there is a defect in JMP 12.1 where the Rao-Scott correction wasn't applied properly. This defect has been corrected in JMP 12.2 which will be released in January.
b. When no overlap exists, but a column is defined as multi-response, the overlap formula is used when it should not. So you get different results when defining a column as multi-response versus not when no overlap does exist.
The multiple response property is how JMP knows to apply the Rao-Scott correction. It should not be applied to columns which are not multiple response types.
Adding to my last message:
If matching the results from JMP to what is implemented in Desktop Reporter becomes an objective, the attached test dataset with results could be used as reference. It contains CATA data as well as multiple product evaluations per consumer. The CATA variable is called Vitamin/Dietary Supmnt Prod N Future Use. In Excel tabs 2 and 3, where this variable is placed “on top” of the table, you can see that the overlap formula is applied.
Thanks,
Alex
We prefer to use the same implementation in JMP as in Desktop Reporter which I guess means the "effective base" calculation. This is driven mostly to avoid questions why the results would differ otherwise. We are not that familiar with the difference between the two options to provide better input than this.
The calculation in the link is weighted average of the standard errors of the means being compared. That makes sense, and I will implement that way.
Do you prefer the "effective base" calculation, which uses the ratio between the squared sum of the weights and the sum of the squared weights, or would you prefer just the weights in the denominator.
Using the weights as the denominator makes more sense to me.
This would make the degrees of freedom for the t-test = w1 + w2 - w0 - 2, where w0 is the count for the overlap. Do you concur?
Since we are no longer pooling standard deviations from means comparisons, it looks like the formula for the variance of the difference between two column means Col 1 and Col 2 with overlaps reduces to the following:
(s1^2+s2^2)/2*[1+P]
where s1^2 and s2^2 are the variances of the means from Col 1 and Col2, respectively.
P is the 2*overlapCount/(n1+n2)
where n1 and n2 are the sample sizes for columns 1 and 2
My edition of Kish makes no mention of the case where the sample sizes are not equal. Are you expecting a weighted average between s1^2 and s2^2?
We do not correct for overlaps on the means test in the Categorical platform.
If you have the same person comparing two products, there are many ways to calculate an appropriate comparison outside of the categorical platform.
Jeff,
Thanks for providing answers to the questions above. It is very helpful for us to understand the capability and limitations of the JMP Categorical Platform. From the response, we understand that this platform should not be used with data where we have multiple product evaluations or multiple time point assessments per consumer.
The only follow up question I have is the following:
Our understanding is that JMP has NOT implemented Kish formula for overlapping samples (sec. 12.4, p. 437 of the book Survey Sampling (1965) by Leslie Kish) in JMP Categorical Platform. That formula has been used in Desktop Reporter to analyze overlapping samples when they occur from sequential monadic test, i.e.study with two products per consumer evaluated in sequential order. Is that correct?
Thanks,
Alex
