nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Conceptual question about validation

dlehman1

I'm hoping that people with a better statistical background than I have can shed some light on this question. JMP has robust validation capabilities for all predictive modeling, including cross validation (though not in the Fit Model platform, except for the capability of using a validation set). Data scientists routinely partition their data into training and validation (and sometimes test) data sets. However, most published work using traditional statistical methods (multiple regression, logistic regression) do not use any validation. I've been wondering why that is the case. Among the possibilities I can think of: (1) statistical significance is viewed as making validation unnecessary; (2) historical practice that is slow to change; (3) the emphasis is on inference and not prediction (along the lines of Brieman's two cultures). In any case, I don't understand any justification for not using validation, so I'd like to hear what people think.

Accepted answers

All comments

Jed_Campbell

I suspect much of the lack of validation comes from your second option, especially in conjunction with other practices/traditions that are slow to change. If we accept the notion that academia tends to only publish significant findings, and if we accept the adage of "publish or perish," then we are naturally disincentivizing, at least to some small extent, extremely rigorous understanding before publication. This might explain some part of the recent discovery that many scientific studies can't be duplicated. In my opinion, if the scientific community adopted validation more readily and was more willing to publish "mundane" findings, I would expect to see results that are more easily duplicated, but perhaps less exciting.

I could easily be wrong or guilty of over-generalization, though. Like you, I'm interested in hearing others' thoughts.

statman

First please define what you mean by validation. One take might be to collect a data set from some inference space and partition it randomly(?) so that only part is used to develop a model and the rest is used to see how well that model predicts the data from the held out data. While this may be accepted practice, this is not my idea of validation (I may be alone in this thought).

One belief I have is to truly “validate” the model I must make physical “samples” and evaluate how well the model predicted those actual samples. To me, validation is how well the model predicts the future (changing conditions). If you have designed your study well enough, over a large enough inference space (which can be very challenging and resource intensive), then your model should perform better in future prediction.

dlehman1

I mean the first definition (the one you don't like), and I think that is what the literature refers to as validation. Of course, almost all models are eventually used for out-of-sample prediction and the partitioning will not address future changing conditions. Nothing will (at least not perfectly). Ultimately, your second idea is necessary - even if you have "validated" a model in the first sense, you should monitor its performance to see if it works well enough to continue to use. But most published papers (other than data science journals) don't even do the first kind of validation, and I am asking why.

P_Bartell

I concur with everything that @statman and @Jed_Campbell have contributed. The only thing I'll add is maybe there is a fourth 'reason'? Until model validation is taught as a requirement of the extended problem solving process of which modeling is a component you just won't see much of it included in the literature. Just another piece or consequence of the 'historical practice that is slow to change.'

SDF1

Hi @dlehman1 ,

Thanks for posing an interesting conceptual question. In general, I agree with what everyone has commented on so far -- that it's important to define what "validation" means in this case. However, I would say that validation can take on both of those meanings -- a hold out set to improve model prediction as well as generating new samples that test the predictive capability of the model.

To me, both are important and need to be done -- one for improving the model, and the other to make sure the model can work, especially when generating new samples that are near the boundaries of the parameter space where models tend to be less accurate.

As to why published works tend not to use validation is to a large extent what you and @Jed_Campbell commented on: historical practice is slow to change -- but this is more of a result of influences within the scientific community that push toward finding a significant result rather than a robust model. My background is physics, and physicists are notorious for creating simple models for one situation and then extrapolating that out to other situations. Take the simple pendulum as an example. This is the basis for almost all basic physics problems, yet in practice it isn't the greatest model -- there have been so many tweaks and changes to it in order to have the model work in other non-ideal situations (think about having to add the electron spin into the orbital mechanics of the electronic states of an atom).

To me, this simple pendulum doesn't make a very robust model -- but it does satisfy the interest and desire to "find" something significant in the data. Sure, it's a great start, and we all need to start somewhere. However, if it comes to wanting to generate a broader, more robust model that can be utilized in more generalized areas, validation is required in both definitions that were discussed.

More "mundane" results might ultimately lead to more robust, better models and predictive capabilities in science, but cultural changes in the scientific community need to change, and a de-emphasizing of the splashy, flashy new findings probably needs to happen.

In short, I would say that not only is historical practice slow to change, but there also needs to be a cultural change in the scientific community about what/how results should be recognized in science. A sort of reorienting of priorities.

statman

I have this to offer:

“The literature as it has grown up seems to be unbalanced in its comparative neglect of the Scientific aspects of the problem, and of its Logical aspects. This perhaps might have been expected, since many of the authors, albeit talented mathematicians, have evidently never submitted their minds to the specifically educational discipline of any one of the Natural Sciences, have never taken personal responsibility for experimentation at ground level, and have no direct experience of the kind of material involved…”

Sir Ronald Fisher, Colloq. Int. Cent. Nat. Recherche Scientifique, Paris, No. 110: 13-19 (1962)

dlehman1

This is a common viewpoint I have seen and I certainly agree with it in spirit. But I think it is inadequate when dealing with the social sciences. Many of the issues (the effects of minimum wages, competition policy, psychological effects of social media use, etc.) are not amenable to scientific experimentation in the same way as the physical sciences. That doesn't excuse people from ignoring good scientific practice, but it does leave a gap (a large one, in my opinion) regarding how to proceed in such areas of inquiry. And I think this does relate to the issue of validation - in my mind it makes the first sort of validation (random partitioning) more essential in these areas (though not sufficient as you have indicated).