Philip Ramsey, Data Scientist, Statistical Consultant and Professor, North Haven Group and University of New Hampshire
Marie Gaudard, PhD, Data Scientist and Statistician, Predictum
Rosemary Danaher, PhD Student in Statistics, University of New Hampshire
Christopher Gotwalt, PhD, Chief Data Scientist, JMP
Artificial neural networks (ANN) and their deep learning can generate highly accurate predictive models. However, they carry with them a hidden risk of scientific disaster: ANNs are very sensitive to the spatial distribution of the input variables. For data with clumps or sizeable gaps, ANNs tend to fit well within the clumps but predict gibberish in the gaps. We use case studies and simulations to show the dangers of applying ANNs to data with poor spatial distribution. We also demonstrate that improvements in predictive accuracy on new data are obtained by designing validation and test sets with high generalizability performance explicitly in mind. By using the popular Pytorch and Tensor Flow ANN libraries, in addition to the algorithms that are native to JMP Pro, we demonstrate that the problem of poor neural network generalizability exists independently of the neural network implementation.
