Permutation Simulation Fail

Question

Hello,

I am trying to perform a permutation test across 6 categories and have followed the instructions here https://www.jmp.com/support/help/14-2/conduct-a-permutation-test.shtml

to do so. The ANOVA I have computed is attached. In step 6 of the Perform the Permutation Test Steps JMP is not allowing me to Simulate. When I click simulate I get an error message "Unable to perform simulation: No columns to switch in. Columns to switch in must have a formula." I am positive that I am right clicking the F ratio as the instructions say and so have no idea what is the issue. Please help!

Best,

Bianca

jules · Answer

Hi @Binoink,

It sounds like you're right-clicking the correct statistic, but that you don't have a formula column to substitute in for an analysis column. The way Simulate works is to have a formula that can be re-evaluated each iteration, and the selected statistic re-calculated upon each run. In the case of the permutation test ANOVA, a shuffling of the responses randomly to generate an empirical sampling distribution of F under the null hypothesis. In the instructions that you linked to the creation of this column is described in the first section, Define the Simulation Formula. This step cannot be skipped, but in fact, can be done even more simply now using Instant Formulas. Simply right-click your Y variable in the data table > New Formula Column > Random > Sample with Replacement (randomization test) or Sample Without Replacement (permutation test). For instance:

Next, run your ANOVA as you did before, and when you Right Click > Simulate, you can specify the Resample[Y] (or Shuffle[Y] if you sampled without replacement) to be substituted in for your original Y. This will return a new data table of the simulated statistics, as well as a data table script (Distribution) which will show the empirical sampling distribution of your statistic (in your case, the F), the original sample estimate indicated by the vertical line, and the empirical p-value for your original sample.

I hope this helps!

@jules

Mark_Bailey · Answer

The scatter plot looks like the data is highly skewed. Is that why you want to use a permutation test?

I think that you want to select Bootstrap for the permutation test, not Simulate.