GenReg for variable selection IssueOne of my customers is using GenReg for variable selection and has been experimenting with Lasso and Elastic Net.
Their model has 14 main effects, with their associated quadractic and interaction terms (lots of variables to examine).
After running Lasso or Elastic Net the final model created contains all terms with no terms zeroed out. The result is strange especially considering that when they run the same analysis using stepwise there are terms ommitted from the model.
Has anyone encountered this type of behavior before when using Lasso or Elastic Net?
Re: GenReg for variable selection IssueThank you all for your suggestions. After a debug session with the customer and @jiancao we were finally able to make sense of their issue which was largely related to using too much data (250K rows) to train the model. The unique characteristics and volume of data was driving the model error to very small levels which then created the situation where every term started to show some level of significance (no terms zeroed out).
By using a smaller, equally representative dataset (15K rows), we were able to get Lasso and Elastic-Net to perform variable selection and build an useful model for their application.
Re: GenReg for variable selection IssueHey Nick,
though they probably have thought thougroughly about validation, is it a good thing to do in their data with random validation? May they have categorical values where you may want to stratify the validation probably by one of those categorical values.
The next thing is trying as Pete suggested Forward selection in GenReg to see if this makes a difference to Lasso in selecting variables. May you also want to use two stage lasso and two stage forward selection.
How does the solution path look like, are there model effects shrinking towards zero (just not hit 0) or not at all (more horizontal lines)?
Finally can you ask the customer to check Bootstrap Forest run multiple times and check if the variables are still all important, or if some of them are once important, the other run not. So the order is not consistent (small changes within the top few is ok, but huge changes are critical). This will tell you that there is something going on in the data which has to be found out before further modeling.
Last but not least, are you using adaptive lasso or not? When your data are highly collinear (would be seen by BF changing orders like crazy 
), the adaptive versions of Lasso and Elastic Net might not provide good solutions. This is because the adaptive versions presume that the MLE provides a good estimate. Uncheck the Adaptive option in such cases.
Best,
Martin
Re: GenReg for variable selection IssueHey Nick,
Have they tried forward selection or backward elimination from GenReg? If they were seeing Stepwise elimate variables they should see that happen in GenReg.
Does the Model include effects with high p-values?
Something else for them to check in the Elastic Net is the Elatic Net Alpha if that is closer to 0 instead of 1 it will work more like a ridge regression and not actually perform variable selection. As far as a Lasso I would just check to make sure they did not force terms to be included under the advanced controls.
Re: GenReg for variable selection Issue150K rows
Re: GenReg for variable selection IssueHow many rows?Re: GenReg for variable selection IssueYes, they are using a validation column.
Re: GenReg for variable selection IssueAre they cross-validating in Genreg?
