Hello,
we have a Pharmaceutical company which need some documentation from us (wite paper, scientific publication, ...) to justify the modeling technique they use (depending on outputs they used Lasso, forward selection, two stage forward selection). For the moment I suggested that selection based on AICc is the right justification but wondering if we have some documnetation that may help them.
Emmanuel
@clay_barker
For using the AICc there is the excellent book by Burnham and Anderson "Model Selection and Multimodel Inference". In the book, they explain how AICc is a very versatile guide for decisions about which variables/effects to include and also to guide decisions about which response distribution to use. They have the book, which you can order, but the authors also have freely available monographs that convey the same information.
One thing that is very underappreciated is how classic procedures, especially p-values, in particular are a very poor guide for model selection. The reasons for this are numerous. One is that the p-values make the critical assumption that the model being tested is the single one that was intended to be used when study was planned. Any selection procedure, from the Lasso, to forward selection, to manual backward selection by eliminating terms with 'large' p-values, violate this assumption as soon as the first change is made to the model. Once a single change has been made to the model using a selection algorithm, the behavior of the p-values is no longer defined.
Another major problem is that p-values cannot be used to compare models that are not ordered, in that one model is a simplification or special case of the other (like one model being the same as the other, but with certain coefficients set to zero). So a model with two main effects and one interaction between them cannot be compared using p-value based methods to a model with three main effects and no interaction, since one model doesn't contain the other.
A very important and underappreciated deficiency is that you cannot use p-values to compare different distributions. For example, if the data are strictly positive, you should consider the gamma and lognormal distributions, in addition to the least squares fit. The 'positive-ness' of the the response means that there mathematically must be curvature terms that are active where the mean is small, in a least squares fit, but these non-normal distributions can handle the boundary at zero without necessarily needing curvature. So you can often get a better fit with fewer terms by fitting models with different response distributions. Using the AICc (and or the BIC) to guide the decision regarding which distribution the response has and the effect that are important is sound statistically, and easy to for the practitioner to take advantage of with a minimum of training.
This last issue of choosing distributions is important, and requires a couple side comments. One is that the response distribution *must* be decided in parallel with effect structure. It is a *major* error to use the Distribution platform to decide the distribution first, ignoring the effects, and then go to GenReg to decide the effect structure based on Distribution's recommendation. Clay Barker has Discovery presentations on this. For example, suppose you have a two level factor, and there really is a difference in means between the levels. The Distribution platform will tell you that the response distribution is a Gaussian 2-mixture. But, once the factor is incorporated into the model, it's likely that a Gaussian (non-mixture) distribution is the right response.
The second aspect of this is the use of distributions as a more principled way of solving the problem that used to be decided by transforming the response. In the old days you tried modeling on the natural scale, then modeled on the log-transformed scale, and then made some kind of decision based on diagnostic plots and what not. You couldn't compare the SSEs of the models on the two scales because they were in different units, which meant that the RSquares were also non-comparable. On the other hand, it is perfectly principled and fair to use forward selection and the AICc on the normal distribution (which is another way of saying least squares), and then do the same thing using the logNormal distribution, and make the final comparison using the AICc.
Anyway, a lot of these concepts are made clear by Burnham and Anderson, as well as their reasoning for rules of thumb for comparing AICc's. The book is intended for people with a science or engineering degree. The authors are ecologists. So, it should be understandable to a large proportion of JMP users, and isn't an abstract tome of arcane theoretical statistics.
Maybe to add on the topic, they analyze their DOEs with GenReg and for their customers and authorities they need to justify why they used what method for this DOE analysis, as Emmanuel said, e.g. using Lasso for a RSM DOE. Why did they use Lasso over Forward or Elastic net.
Is there any publication one could refer to so they can submit this paper reference together with their analysis?
@clay_barker @chris_gotwalt1
