Assessing Models in GenRegHi,
When building a model using Generalized Regression, if I use Standard Least Squares -- with or without a Validation Column -- then JMP outputs a number of useful metrics under the Model Summary outline. However, if I use a different method -- Forward Selection or Lasso or whatever else -- so I can get at other validation methods besides the validation column, there are no single metrics that I can compare. So if I have a validation column, I can get something like an R^2 of the training set and of the validation set. But if I use leave-one-out, or AICc, or a holdback set, or anything else, I can't get any good overall model summary statistics. I assume the solution path plots are about the best I can get.
The solution path plots aren't especially helpful, though, if I am trying to compare multiple approaches. If I go back to traditional stepwise, I can run the same set of candidate terms through AICc selection and BIC selection, see that I get different models, and compare the R^2 or the RMSE or the AICc or the BIC or some other model summary statistic to see which model is "better". If in traditional stepwise I choose a K-Fold cross-validation I can get a "RSquare K-Fold" statistic.
What are the analogues for these tools in Generalized Regression? I'm really excited to take advantage of the flexibility that GenReg can offer but I'm at a loss as to how to choose between various options. I'm sure there's something simple, but being relatively new to GenReg I'm just not sure what it is.
Thanks,
Michael
Re: Assessing Models in GenRegThanks for working with us on this and please make sure to add what you would like to see to wish list.
Best,
Bill
Re: Assessing Models in GenRegYeah... I'm working with some smaller data sets, which is why I'm leaning towards the Leave-one-out, but that brings me back to my frustration of trying to assess the model. I'd love something a little more quantitative that explains why JMP arrives at a particular model, rather than just pressing a Go button and hoping for the best.
Re: Assessing Models in GenRegMichael,
I believe you can accomplish what you want with a 3 level validation column. Training, Validation, and Test. The Test data is not used in the model fit and the RSquare from the Test set is then compared to the RSquare for your Training and Validation sets. The closer, the better. This is a little tougher to accomplish with a smaller data set (< 100 rows), but I have found it to be a very good way to determine how well your "all in model" will predict future data.
As to your last point, I believe Leave-one-out K-fold is one of the best cross-validation techniques you can choose for smaller data sets.
Re: Assessing Models in GenRegOK Bill,
Thanks. I am still learning but I think I understand a little more now.
I think what I'm getting at is that an all-in R^2 (on all rows of the data) is useful, but I am looking for something more like this:
The difference between the model fit statistics for the training set vs. the validation set gives perspective on how robust the model is to observations that are withheld from the fitting process. One of the things I like to look for when building models is some statistic that tells me that if I were to get more observations that my model would still be appropriate.
Perhaps the only way to do this is to create a validation column and randomly identify one set of observations to withhold, rather than trying to do a multi-fold validation (identifying multiple sets of observations to withhold). My thinking is that mullti-fold validation would lead to a more robust model, but perhaps that's not true?
Michael
Re: Assessing Models in GenRegMichael,
I had further correspondence with the developer based on your reply and below is what he sent back;
"...what he’s comparing is the R^2 in model comparison (for a forward selection model built in genreg) to the R^2 of a forward selection model built in the stepwise platform. The R^2 in his model comparison (.8704) is calculated on the entire data. The Stepwise output actually includes that information as well, I’ve circled it in red below. The value that he circled (.8514) in the Stepwise report is the is the k-fold R^2. So the issue is that he’s comparing an R^2 on the entire data to an R^2 for k-fold." 
The developer goes on to talk about differences in the two platforms;
"....(when) comparing forward selection in genreg to forward selection in Stepwise. There are a few subtle differences in how the two platforms do forward selection. Genreg looks at score tests to decide which effect to bring in next, Stepwise looks at wald tests."
And finally the calculation for K-fold RSquare in Stepwise from Fang Chen in the development group:
"The K-Fold RSquare in Stepwise is calculated by following the general definition of RSquare = 1 – SSE/SST, where SSE is calculated based on residuals from each of the K holdout samples.
K-Fold RSquare = 1 - ResV`*ResV / ( (Y-YMean)`*(Y-YMean)"
This last bit may be the crux of what you were looking for as to why the RSquare values are different. I added the other bits in the hopes of providing a more complete answer.
HTH
Re: Assessing Models in GenRegBill,
I'm still confused. The GenReg developer says that GenReg is doing a comparison of the model (R^2? something else?) but I don't know where to find it, see my original post saying I couldn't find any metrics. Also interesting that he says stepwise and model comparison calculate R^2 in the same way. That's strange since I clearly got two different values for R^2.
Thanks,
Michael
Re: Assessing Models in GenRegMichael,
Please see the explanation below from the one of the developers of the GenReg platform.
"When we calculate R^2, we’re comparing how well our model fits compared to a null (intercept-only) model. On the training set, it is obvious how we should do this – we compare our fit to the intercept-only model estimated on the training set. On the validation set, there is some ambiguity. In genreg, we compare the model that we fit to the intercept-only model built on the training set. In stepwise and model comparison, the R^2 is calculated comparing the fitted model to the intercept-only model built on the validation set."
The developer went on to say that he feels "one answer isn't necessarily right or wrong". That said though, he strongly prefers the way it is done in GenReg. 
Re: Assessing Models in GenReg@Bill_Worley Thank you! On the surface this is a useful approach. It does disagree with what I get in Stepwise. Which platform should I believe? (GenReg with Forward Selection, Leave One Out, 38 rows of data in the table, vs. Stepwise, k-Fold from red triangle with 38 folds specified.) See screenshots. Fairly noticeable differences in R^2 and minor differences in coefficient estimation.
Re: Assessing Models in GenRegHey Michael,
My first reaction to your question is to suggest using Model Comparison. Publish the prediction formulas you want to compare to the Formula Depot and go from there or go directly from the Gen Reg report to Analyze > Predictive Modeling > Model Comparison. You will have to save your prediction formulas to the data table first to use this option though.
HTH
Best,
Bill
