Level: Intermediate
Philip J. Ramsey, Senior Data Scientist, Consultant and Professor, North Haven Group and University of New Hampshire
Trent Lemkus, PhD Candidate in Statistics, Department of Mathematics and Statistics, University of New Hampshire
Christopher Gotwalt, JMP Director of Statistical Research and Development, SAS
The primary goal of designed experimentation (DOE) is often prediction of future system behavior, yet historically no direct method to assess model prediction performance existed. The prediction validation strategies used with big data are infeasible with DOE. Gotwalt and Ramsey (2018) introduced a new predictive modeling strategy for DOE called fractionally weighted bootstrapping with autovalidation (FWB+AV). The method extends machine learning type model selection strategies that take advantage of holdout cross validation to small data sets where it is not possible to hold out even a single row for independent assessment of model fit. Autovalidation exploits a unique weighting scheme and bootstrapping concepts to determine the subset of experimental effects most important to prediction. Despite it being a very recent development, the authors are already seeing it put into practice at biotech and pharmaceutical companies in the US and Europe. This has expedited the importance of evaluating the performance of FWB+AV relative to other competing approaches to model selection. Using innovative simulation techniques we demonstrate that FWB+AV outperforms virtually all commonly used predictive model building techniques used with DOE, as measured by predictive accuracy of the final model used. In this talk we discuss the simulation results and show how to easily perform FWB+AV with JMP Pro. Additionally FWB+AV will be demonstrated live using actual case studies. FWB+AV allows powerful machine learning algorithms, used with big data, to be applied to DOE modeling.
