Question about how to correct for multiple comparisons in ANCOVA (BY=Atlas) resultsHello, I am currently using JMP Student Edition to perform an analysis of covariance (ANCOVA). Here are the steps:
- Model fitting (Y: measured value, BY: atlas number)
- Model effect configuration (disease number, age)
- Method: Standard least squares
- Emphasis point: Here's the effect
When you run this setting, a report of "Actual response values" will be displayed for each atlas number.FDRI understand that checking this box applies FDR correction.
Purpose: We tested statistical significance for each atlas number using ANCOVA, but because we performed separate tests for each atlas, it is possible that false positives may be present. Therefore, we would like to correct the obtained p-values for multiple comparisons across all atlases.
Here is the question:
- Does the FDR correction in the "Effect Summary" only apply to the number of factors included in the model effects (in this case, the two factors: disease number and age)?
- Is it possible to perform FDR correction for ANCOVA results across all atlas numbers (all BY reports) in JMP Student Edition? It's not possible via GUI or JSL script.
- *I would appreciate it if you could answer this question in the case where the FDR correction in ① is only applied to the number of coefficients of the model effect.
Advanced Statistical Modeling
Re: Question about how to correct for multiple comparisons in ANCOVA (BY=Atlas) resultsThe "FDR" in the "Effect Summary" displayed when using a By variable in "Fit Model" is the FDR-adjusted p-value for the test within each level of the By variable only.
Note that the FDR (false discovery rate) does not control the FWER (familywise error rate), i.e., the probability that one of the true null hypotheses will be rejected. It is merely an FDR-adjusted p-value calculated using the Benjamini-Hochberg method to control the average proportion of true null hypotheses among rejected null hypotheses.
■A function called FDR Adjust has been added in JMP 19. Although it requires programming, this function allows you to calculate an FDR-adjusted p-value from a raw p-value (unadjusted p-value). The steps are as follows:
Please note that the script below is only a sample, and I am solely responsible for this script, not my organization.
(1) Run a regression analysis using Fit Model .
(2) Right-click on the "Parameter Estimates" table and select "Create Concatenated Data Table" from the pop-up menu. This will create a data table containing the parameter estimates and p-values.
(3) With the data table window created in (2) above active, select [File] → [New] → [JSL Script] . The script editor will be called up.
(4) Copy and paste the code below into the script editor in (3).
Names Default To Here(1);
/* 現在、アクティブなデータテーブルに対する参照を取得 */
dt = Current Data Table();
/* 切片項の行を選択 */
dt << Select Where( :項 == "切片" );
/* 切片項の行を削除 */
dt << Delete Rows;
/* 未調整p値の取得 */
pval = Column(dt, "p値(Prob>|t|)")<
<GET as="" matrix=""></GET>
<P><BR />(5) Select <STRONG>[Edit] → [Run Script]</STRONG> .<BR /> A column called "FDR" should be added to the data table above (2).</P> <P><BR />■Although this involves testing the entire data set rather than just each By level, a new subgroup analysis function has been added to Response Screening in JMP 19.</P> <P>(1) Select <STRONG>[Analyze] → [Fit Model]</STRONG> .<BR /> (2) Select <STRONG>[Response Screening]</STRONG> from <STRONG>[Method]</STRONG> in the upper right corner of the window.<BR /> (3) Specify the response variable (Y variable) and effects.<BR /> (4) Specify the By variable in <STRONG>Subgrouping</STRONG> .<BR /> (5) Click the <STRONG>[Execute]</STRONG> button.</P> <P>The "FDR p-value" in this result is an FDR-adjusted p-value that takes into account not only the test for each By level but also the test for the entire data.</P> <BR /> <P>Again, the FDR-adjusted p-value does not control the FWER, but rather the false discovery rate.</P> <BR />
Re: Question about how to correct for multiple comparisons in ANCOVA (BY=Atlas) resultsThe FDR-adjusted p-values shown in the "Summary of Effects by By Variables" are for each By series (In this case, I clearly understood that it only applies to the test results within the atlas number.
To perform FDR correction across all atlases, use theFDR Adjust functionI found it very helpful that I could calculate FDR-adjusted p-values from unadjusted p-values all at once using.
Thank you for providing the script as well.
Another new discovery was that in JMP19, specifying subgrouping in "Response Screening" makes it possible to perform FDR correction on the entire data set.
I also found your comments about the differences in thinking between FDR and FWER very informative.
Thank you very much for your valuable explanation.
