Is there a JMP Script to run a Cochran's Q Test?
If your case is 2x2, you can use McNemar's test in JMP. Select Analyze Fit Y by X, assign your X and Y colums, click OK, and the click the red triangle and select Exact Agreement.