Can both Pearson and Spearman calculate the P-value from the Fisher z-transform to that P-value?Currently in JMP, Pearson is also Spearman Also that P value is,( z z No conversion required) t It is a value of distribution approximation. But from the Fisher z transform P I am eager to calculate the value. In particular, in recent papers submitted to biological systems, the correlation coefficient Pearson is also Spearman is also common. In that case, the P-value calculation from the Fisher z-transform is overwhelmingly large. I would like to thank you. Also, as for the confidence interval configuration, we would like to have a 95% confidence interval configuration by Fisher z-transform as much as possible. (In addition, the exact method is not required for this case)
Statistics and AnalyticsRe: Can both Pearson and Spearman calculate the P-value from the Fisher z-transform to that P-value?Thank you for your request.
Also, I'm sorry for the delay in replying due to the time it takes to communicate within our group.
My name is Ono and I am a tester at the JMP Japan Division of SAS Institute Japan Co., Ltd.
We received a request to add Fisher's Z-transform approximation method as an option for the p-value for the Pearson and Spearman correlation coefficients. The same method has been added to the request list. We would like to consider it further.
Current " Multivariate correlation On the platform, the p-value for the Pearson and Spearman correlation coefficients is calculated from the Student's t distribution.
For Pearson's correlation coefficient , The p-value based on this Student's t distribution is an exact p-value under the following conditions:
- The distribution of each observation vector is an independent and identical distribution from the multivariate normal distribution.
- There are no missing values.
- The null hypothesis is H0: ρij = 0 (where ρij is Pearson's population correlation coefficient between the i and j variables).
JMP's current " Multivariate correlation On the platform, the confidence intervals for Pearson's correlation coefficient are calculated from Fisher's Z-transform approximation (without the bias term). Therefore, if the confidence interval does not contain a constant c (-1.0 <1.0), it means that the null hypothesis "H0: ρij = c" is rejected (default confidence factor is 95). %, That is, the significance level is 5%).
If the sample size is large to some extent, the test based on Student's t distribution and the test based on Fisher's Z-transform have similar results as a test for the null hypothesis "H0: ρij = 0". However, the results are different, especially for small samples. For this reason, the current " Multivariate correlation On the platform, it is possible that the test and the confidence interval will result in discrepancies (the statistical test may reject "H0: ρij = 0", but the confidence interval may contain 0. The opposite is also possible.) This may be a bit confusing for users. However, the method based on the Student's t distribution described above is an accurate method under the above conditions and is easy to calculate. On the other hand, "accurate" confidence intervals are a bit difficult to calculate. For that reason, I think it is rational to use Student's t distribution as the p-value and Fisher's Z-transform approximation as the confidence interval.
Current " Multivariate correlation The platform does not calculate confidence intervals for Spearman's correlation coefficient.
As far as we know, there are two methods for "accurate" tests and confidence intervals. The first is the confidence interval calculated on the premise of a multivariate normal distribution with respect to Pearson's correlation coefficient. This method requires numerical integration if ρij ≠ 0. Due to the complexity of the calculations, it seems that the exact test for ρij ≠ 0 and the exact confidence interval for ρij, in line with this idea, are not very popular. Another exact test is based on the sort test. The sort test is a conditional "accurate" method. It is difficult to know how to sort when ρij ≠ 0. Neither exact method is implemented in JMP. Only the exact method based on Student's t distribution (when it is a multivariate normal distribution) at "H0: ρij = 0" is provided for Pearson's correlation coefficient.
In summary, the current JMP has the following structure.
- The p-value for the Pearson and Spearman correlation coefficients is calculated based on the Student's t distribution.
- The confidence interval for Pearson's correlation coefficient is calculated by Fisher's Z-transform (no bias term) approximation.
- Confidence intervals for Spearman's correlation coefficient are not calculated.
If you want to calculate the p-value from Fisher's Z-transform approximation on the current JMP, you need to program in JMP Scripting Language (JSL). The code looks like this (the code below is from Fisher's Z-transform with no bias term): If you are interested in calculating Fisher's Z-transform with code like below, please reply. I will explain the code. In addition, the following is to show the calculation method in an easy-to-understand manner, but the code can be made more versatile.
r = 0.4;
n = 15;
z = 0.5 * Log((1+r)/(1-r));
up_z = z + Normal Quantile(0.975)*Sqrt(1/(n-3));
low_z = z - Normal Quantile(0.975)*Sqrt(1/(n-3));
Show((Exp(2*low_z)-1)/(Exp(2*low_z)+1));
Show((Exp(2*up_z)-1)/(Exp(2*up_z)+1));
p_z = 2*Normal Distribution(-Abs(z*Sqrt(n-3)));
Show(p_z);
t = r * Sqrt((n-2)/(1-r^2));
p_t = 2 * T Distribution(-Abs(t), n - 2);
Show(p_t);
Excuse me for the long message.
We would like to thank you again for your request.
If you have any questions or concerns regarding the above reply, please reply.
Re: Can both Pearson and Spearman calculate the P-value from the Fisher z-transform to that P-value?Mr. Ono
Thank you for your contact.
I am not in a hurry, but rather thank you for your polite answer.
Therefore, it is as follows.
(1) Give up the 95% confidence interval configuration with bias correction by Fisher Z transform for Spearman's correlation coefficient.
(2) Regarding the p-value for the correlation coefficient of Pearson and Spearman, among "Fisher Z transform", "Student's t distribution (currently installed)", and "accurate method", P by "Fisher Z transform". Because I want to describe the value in the paper
Regarding the p-value for the Pearson and Spearman correlation coefficients,
1) I am interested in the calculation of Fisher's Z-transform and would like to explain the code.
2) I am interested in the calculation of Fisher's Z-transform and would like a more versatile code explanation.
It will be.
In (2) above I give up on the "accurate method".
The background is because we want to use it for clinical image analysis of 20-30 cases, so it is a small number of 20-30 cases.
I am currently using JMP 15.2 (64bit version of win10)
We continue to thank you.
Re: Can both Pearson and Spearman calculate the P-value from the Fisher z-transform to that P-value?I uploaded the explanation PDF document and the program example to the page of the link below.
Program example (Fisher z-transform)
The PDF document first gave an overview of the JMP scripting language, and then explained how to find the p-values and confidence intervals by Fisher's z-transform approximation. Finally, I presented an example program that requires them more generically.
The file with the extension jsl is an example program. Download this file first, open the data table to be analyzed with JMP, and then open the file by selecting [File] → [Open]. If you select [Run Script] from the menu that is called by right-clicking, the program will be executed. Select the column to be analyzed in the called startup dialog.
If you find any questions or deficiencies, we would appreciate it if you could reply.
