After some investigation by JMP development, it appears that there is a bug for the case when the KFold validation sets are created and bootstrapping is done in the same Generalized Regression report. The random seed used for validation set creation is not getting properly reset for each bootstrap iteration. This bug should be fixed in time for JMP14.
In the meantime, here is a workaround. You can fit your model in Generalized Regression using KFold cross-validation as you did before and then save the validation sets (Save Columns > Save Validation Column). Then launch Fit Model again, click Recall and use the validation column that you saved in the Validation role. Run the new model. If you bootstrap from this new report, you should bypass the bug.
Hope that helps. Let us know if you have further questions.
-Michael
Hello,
Please provide more information about the situation where you are seeing identical results for all bootstrap iterations:
Thanks,
Michael
JMP pro 13.0.0 (64 bit)
several different platforms including Generalized regression
did not choose a seed
Also I have noticed that if I launch an analysis that may have a stocastic feature such as a Cross-fold validation or bootstrapping the first one looks fine, then a redo analysis is slightly but appropriately different, and then after that each redo yields identical results to the second analysis. In both cases it looks like something is not getting reset to allow for a truely random execution whether it be bootstrapping or cross-fold validation.
Yes I can. I used the Nicardipine data set and attempted to model sex as a function of ~10 continuous predictor variables. The issue appears to be linked to using cross-validation as the validation method. For both Adaptive Elastic Net and Adaptive Lasso when I use CF validation and then attempt to do a bootstrap on the parameter estimates I get identical results for boot iterations 1 to n. (boot 0 is different). I tried to model a continuous variable in the same way and found that the first few iterations were different but eventually all the boot iterations yielded identical results? Almost appears to be converging on a solution and then never changing. When I use AIC as the validation method it looks like bootstrap works fine.
@dherring_wakehe wrote:Yes I can. I used the Nicardipine data set and attempted to model sex as a function of ~10 continuous predictor variables. The issue appears to be linked to using cross-validation as the validation method. For both Adaptive Elastic Net and Adaptive Lasso when I use CF validation...
Yes I can. I used the Nicardipine data set and attempted to model sex as a function of ~10 continuous predictor variables. The issue appears to be linked to using cross-validation as the validation method. For both Adaptive Elastic Net and Adaptive Lasso when I use CF validation...
I'm afraid I'm not able to reproduce it with this info.
Thank you for the investigation and reply! I'll use the work around for this admittedly rather esoteric analytic sequence. - DH
