Distributions in GenReg

Question

I asked Chris about distributions in GenReg

In the model launch window for GenReg it says response distribution.  Is this correct?  Or is it the distribution of the residual of the model?

The short answer is that it’s the response distribution and not an error distribution that is being modeled in general.  For the Normal and a couple other distributions like the Cauchy you can think of the error distribution and the response distribution as the same, but they aren’t the same outside of a few special cases.  For example, one never thinks of an “error term” for the binomial, Poisson, Lognormal, Gamma, or Weibull distributions, just response distributions.  It’s only the cases where the distribution is symmetric around the mode where one considers there to be a “y= mean model + error term” concept that applies.   What happens is people learn least squares modeling first using the model+error concept, but then to really understand Logistic and Poisson regression you have to unlearn it and think of it using response distributions.  Unfortunately, outside of certain university settings, this point is almost never made clear.  The good news is that, for the most part, this subtle lack of understanding doesn’t get in the way of people using the models successfully in industry.

Confusion about this totally understandable because it is very widespread.  Even the opening sentence in the Wikipedia page on Generalized Linear Models gets it wrong when they say:

“In statistics, the generalized linear model (GLM) is a flexible generalization of ordinary linear regression that allows for response variables that have error distribution models other than a normal distribution.”

They do begin to get it right in the second sentence, however…

What is really going on in Logistic, Poisson, or Gamma regression is that we are allowing the response variable to have a non-normal distribution.  The relationship between the response and the predictor variables occurs by placing a linear model on a suitably chosen function of the mean, which is called the link function.  In the case of the Logistic (Binomial) regression this is a logistic() function, in Poisson it is the log(), etc.  Combining this mean model with a non-normal distribution for estimation allows us to make more efficient use of the data because it inherently takes into account how the variance changes as a function of the mean, which is the case for all non-negative distributions and allows us to get the inferential aspects like hypothesis tests and confidence intervals of the analysis correct.

DonMcCormack · Answer

Forgot to say that the saved script will closely estimate the parameter values from Mean Response using N Tries, bringing everything back full circle.

DonMcCormack · Answer

Thanks for bringing this up. It's been something I've had trouble getting my head around well enough to explain what's going on. Your post has given me the motivation to come up with a way of explaining GLMs and GenReg better.

To clarify, the response distribution is not what gets specified in the launch dialog, at least not what I typically think of as a response distribution. It is the distribution of observations for fixed input values and model parameters (i.e., the observations conditional on the predictors). Clay mentions this in his talk on slide 18. This conditioning says, "don't look at the distribution across the range of inputs but for a given input value" (or vector of values when we're dealing with multiple inputs). In my opinion, this characterizes dispersion, what I would call error variance. I also believe that much of the confusion is because the word 'response' is used to not only indicate, in general, the observed unconditional observations but also to indicate, in the case of GLM or GenReg, the observed conditional values, too. To avoid this confusion, it might be clearer to drop the adjective 'response' when talking about the later.

I'm attaching a table of simulated data to help illustrate what is going on. Let's say we're interested in generating an example on predicting the number of failed attempts required to successfully complete a task relative to how many years experience a worker has. The negative binomial (NB) is the standard distribution for counting n failures until the rth success. The formula in the column Mean Response is what we'll condition on. To generate dispesion around the mean response, we'll simulate random NB variates with parameters r = 1 (given as a table variable) and p (which is calculated using the values for r, mean response, and the quadratic formula). If you look at the formula for N Tries, you'll see that it is a function of r (a fixed value) and p (via Mean Response), a value that changes relative to Years Experience, making it based on multiple NBs. You can see this if you simulate a large number of observations for the same years experiences, then fit a NB to the data. You get a dispersion estimate close to 1 (dispersion = 1/r) and a mean estimate of r*(1-p)/p (the expected mean of a NB). Note that the values in Mean Response and the mean of the NB are two different things. The former specifies the linear relationship between (unconditional) predictor(s) and response after the link function is applied. The later is the mean of the dispersion for observations conditioned on fixed input values. For low value of years experience, the diference between Mean Reponse and the NB mean estimate of the can be relatively large. As for the residuals, they are just the difference between the Mean Response and N Tries columns. Given what I mentioned above regarding how N Tries is created, it's not surprising it doesn't follow any one given distribution.

Demo GenReg Details.jmp

wendytseng · Answer

Thank you for posting this, Pete!  I found this confusing as well.  The normality of the residuals assumption was always emphasized more to me than the distribution of the response.