Is it possible to calculate the P value from Fisher z transformation for both Pearson and Spearman's P value?Currently at JMP, Pearson also Spearman Mosono P value is,( z (no conversion required) t This is the value of the distribution approximation. But from Fisher z transform P I am eager to calculate the value. In particular, in recent biological science submissions, the correlation coefficient Pearson Spearman is also often seen. In that case, the P value is overwhelmingly calculated from the Fisher z transformation. I would like to ask for your help. In addition, it is desirable to construct a 95% confidence interval using Fisher z transformation as much as possible. (Please note that the Accuracy Act is not required in this case)
Statistics and AnalyticsRe: Can't you calculate the P value from the Fisher z transformation for both Pearson and Spearman?Thank you very much for your request.
Also, I apologize for the late response as it took some time for communication within our group.
My name is Ono and I work as a tester in the JMP Japan division of SAS Institute Japan.
We have received a request to add an option to approximate Fisher's Z-transformation for the p-values for the Pearson and Spearman correlation coefficients. We have added this to the wish list and will consider it further.
In the current Multivariate Correlation platform, p-values for Pearson and Spearman correlation coefficients are calculated from the Student's t-distribution.
For the Pearson correlation coefficient , this p-value based on the Student's t-distribution is an exact p-value under the following conditions:
- The distribution of each observation vector is an independent and identical distribution from a multivariate normal distribution.
- There are no missing values.
- The null hypothesis is H0: ρij = 0 (where ρij is the Pearson population correlation coefficient between the ith and jth variables).
In the current Multivariate Correlation platform in JMP, confidence intervals for the Pearson correlation coefficient are calculated from a Fisher Z-transformation approximation (which does not include a bias term). Therefore, the confidence intervals are calculated based on a constant c (-1.0 <1.0）を含まないのであれば、帰無仮説「h0: ρij="c」が棄却されることを意味します（デフォルトの信頼係数は95%、つまり、有意水準5%となっています）。
If the sample size is relatively large, the test based on Student's t distribution and the test based on Fisher's Z transformation will give similar results for the null hypothesis "H0: ρij = 0". However, especially in the case of small samples, the results of the two are different. For this reason, in the current " Multivariate Correlation " platform, the test and the confidence interval may give discrepant results (the statistical test may reject "H0: ρij = 0", but the confidence interval may include 0, or vice versa). This may be somewhat confusing for users. However, the method based on Student's t distribution described above is accurate under the above conditions and is easy to calculate. On the other hand, the "exact" confidence interval is somewhat difficult to calculate. For this reason, we believe it is reasonable to use Student's t distribution for the p-value and Fisher's Z transformation approximation for the confidence interval.
The current " Multivariate Correlation " platform does not calculate confidence intervals for the Spearman correlation coefficient.
As far as we know, there are two methods for "exact" tests and confidence intervals. One is a confidence interval for Pearson's correlation coefficient, which assumes a multivariate normal distribution. This method requires numerical integration when ρij ≠ 0. Due to the complexity of the calculations, it seems that exact tests for ρij ≠ 0 and exact confidence intervals for ρij are not widely used. The other exact test is based on a permutation test. The permutation test is a conditionally "exact" method. It is difficult to know how to perform the permutation when ρij ≠ 0. Neither exact method is implemented in JMP. The only exact method provided for Pearson's correlation coefficient is a Student's t distribution (when the distribution is multivariate normal) when "H0: ρij = 0".
In summary, the current structure of JMP is as follows:
- The p-values for Pearson and Spearman correlation coefficients are calculated based on the Student's t-distribution.
- Confidence intervals for the Pearson correlation coefficient are calculated using an approximation of Fisher's Z transformation (without the bias term).
- Minitab does not calculate confidence intervals for the Spearman correlation coefficient.
If you want to calculate the p-value from Fisher's Z-transformation approximation in the current JMP, you need to program it in JMP Scripting Language (JSL). The code is as follows (the code below is Fisher's Z-transformation without bias term). If you are interested in calculating Fisher's Z-transformation using the code below, please reply. We will explain the code. Also, the following is intended to present the calculation method in an easy-to-understand manner, but it can also be made into a more versatile code.
r = 0.4;
n = 15;
z = 0.5 * Log((1+r)/(1-r));
up_z = z + Normal Quantile(0.975)*Sqrt(1/(n-3));
low_z = z - Normal Quantile(0.975)*Sqrt(1/(n-3));
Show((Exp(2*low_z)-1)/(Exp(2*low_z)+1));
Show((Exp(2*up_z)-1)/(Exp(2*up_z)+1));
p_z = 2*Normal Distribution(-Abs(z*Sqrt(n-3)));
Show(p_z);
t = r * Sqrt((n-2)/(1-r^2));
p_t = 2 * T Distribution(-Abs(t), n - 2);
Show(p_t);
Sorry for the long post.
Thank you again for your request.
If you have any questions or concerns about the above reply, please feel free to reply.
Re: Can't you calculate the P value from the Fisher z transformation for both Pearson and Spearman?Mr. Ono
Thank you for contacting us.
I'm not in a hurry so thank you for your prompt response.
In this regard, the following applies.
(1) We will give up on constructing a bias-corrected 95% confidence interval for the Spearman correlation coefficient using the Fisher Z transformation.
(2) Regarding the p-values for the Pearson and Spearman correlation coefficients, out of the three methods available, Fisher Z transformation, Student's t distribution (currently available), and the exact method, I would like to use the Fisher Z transformation p-value in my paper.
Regarding p-values for Pearson and Spearman correlation coefficients,
1) I am interested in calculating Fisher's Z transform and would like an explanation of the code.
2) I am interested in calculating Fisher's Z transform and would like a more versatile code explanation.
It will be.
At (2) above , we give up on the "exact method."
The background is that we want to use it for clinical image analysis of 20-30 cases, so the number of cases is small, 20-30 cases.
I am currently using JMP15.2 (64-bit version for Win10).
Thank you for your continued support.
