446912_en-US

liliconcom

JMP > Identification of predictive signature from large data set > Issue with obvious overfitting

Hi JMP community,

I have a large data set with ~4,000 biomarker measurements at Baseline in patients enrolled in a clinical trial. I need to identify among this data which combination of biomarkers may predict the clinical outcome 12 weeks later.

I have used a rather naive approach that appears to yield grossly overfitted outputs:

1. Run all 4,000 biomarkers through the Screening > Response Screening where Response is the binary clinical output at week 12
2. Select the top 10 - 20  biomarkers (based on FDR)
3. Use the selected biomarkers (as Model Effects) on the Fit Model > Stepwise platform where Y is the binary clinical output at week 12; criteria = minimize BIC
4. Generate the model

While this approach yielded excellent "predictors" to the clinical response at week 12 (ROC AUC > 0.940), it also yielded high ROC AUC when tested with a randomly generated "dummy" variable (ROC AUC ~0.870). The latter indicates to me that my method is grossly inappropriate because of overfitting.

I explored the feasibility of working with  Discovery and Validation patient subgroups (50% of the entire population), but it did not solve my issue with overfitting.

 

Besides using the advanced modeling platforms provided by JMP Pro, are there better techniques in JMP to approach such a question?

 

Thank you for your help.

 

Best,

TS

 

 

AnalyticsRe: JMP > Identification of predictive signature from large data set > Issue with obvious overfitting

Can you provide some more detail about your randomly generated response variable?  How exactly did you create it? 

Re: JMP > Identification of predictive signature from large data set > Issue with obvious overfitting

Hi Dale,

 

Thanks for following up on this relatively open question. I created the random variable using the following formula:

random uniform (0,1) > 0.5;

Of note, I ran my "method" described above on eight different versions of the dummy variable (i.e., recalculate formula), yielding ROC AUC values ranging from 0.770 to 0.870.  

I never came across such a behavior, but I have never worked with more than 1,200 biomarker variables.

Thanks for your help.

Best,

TS

Re: JMP > Identification of predictive signature from large data set > Issue with obvious overfitting

I don't know the answer, but I can confirm what you found.  I created a binary response variable (random) and 20 random normal factors with 200 observations.  I got AUC=.71!  I find it very surprising indeed.  I suspect it has something to do with a limitation of the AUC as a performance measure, but as I said, I don't really understand this behavior.  In my case, it certainly should not be overfitting.  I'm anxious to see what others can say about this.

Re: JMP > Identification of predictive signature from large data set > Issue with obvious overfitting

To add to the mystery, I've attached the simulation I did.  This time I used a validation column.  Things worth noting:  the logistic regression (script saved in the file) shows AUC around .71 for training data and much lower (less than .5) for the validation data.  That is clear evidence of overfitting.  However, I also ran the model screening - all the methods (including the logistic regression) show AUC around 0.5 for the training data and lower for the validation data - that seems reasonable to me.  In addition, the logistic regression model is not significant (and only 1 or 2 factors are significant, about what you'd expect for 20 random factors).  Among the things I don't understand is why the logistic regression platform and logistic regression within the model screening platform give such different results.  I know (or believe) that model screening does not find tune hyperparameters as much as the individual modeling platforms, but the difference between 0.7 and 0.5 on the training data seems too large to me.