Ridge penalty function question in Generalized regressionOur customer have some questions about the Generalized regression, according to their experience before, they want to use “Ridge” as the penalty function to construct the model because they don’t want to miss any variable’s effect, the “Elastic Net” or “Lasso” penalty function are not fit in this case.
However, customer usually analyzed spectrum data with many variables(some data contain 4500 variables with 65 rows), it took long time and can’t get the result when e use “Ridge” as the penalty function.
I had feedbacked the question to JMP support, they try the sample I gave( as attachment, around 2400 variables and 65 observations ) on Windows 10 machine, with 16GB RAM and 4 cores. And JMP went into "Not Responding" mode for the first hour, but then produced a progress bar. It took long time to get the ridge result(4.5hr), even change the grid point to 50.
Compare with Elastic net or Lasso method, those run about 2~3 minutes, there is big time spend difference between Elastic net/Lasso and Ridge
Moreover, customer also mentioned that Python can execute the Generalized regression with “Ridge” as the penalty function more quickly.
Is it possible to speed up the model fit process? Something like making CPU full loading? Or other suitable feature/platform can help customer to get estimate result more efficiency?
I attached the sample data as well. Thank you
Re: Ridge penalty function question in Generalized regressionFWIW, I have a Windows 10 box with 32 cores and 128GB of RAM. I'm not sure what JMP will take advantage of, but I'd be happy to try to get another benchmark if it helps. But the details matter (particularly if it takes five hours for a run . . . ). So could you post the script to do the ridge regression, please? I didn't see it in the table.
Re: Ridge penalty function question in Generalized regression@Eric_Wang Some implementations of Ridge Regression in Python require input variables to be standardized. Standardizing the variables could speed up the time to solve for the coefficients. I'm not sure if JMP's routine automatically performs some kind of standardization for the inputs but you can experiment if standardizing the ~4500 input variables allows for a quicker compute time.
Re: Ridge penalty function question in Generalized regressionThis sounds like a case where we should be using the elastic net. In cases like this where there are far more predictors than observations, the big concern when doing variable selection is that the collinearity and redundancy will cause us to miss predictors that truly matter. If X1 and X2 are highly correlated, something like forward selection is likely to only pick one of them. The strength of the elastic net is that it will select groups of correlated predictors, so X1 and X2 in this case. My guess is that in this situation "don’t want to miss any variable’s effect" actually means that they do not want to miss any effects that might have a real impact. The elastic net is built for this exact scenario.
So why are lasso and elastic so much faster than ridge regression here? Because they do variable selection, elastic net and lasso computations only involve a handful of predictors while ridge regression is doing optimization over all 2400 variables specified. It's also important to note that lasso and elastic net both tend to dominate ridge regression in terms of fit quality and interpretation, so we've made efforts to improve computation times for lasso and elastic net fitters. We have not been able to prioritize similar improvements for ridge regression.
Re: Ridge penalty function question in Generalized regressionThanks, Ian, I modified and reattached the sample data with the script in script area of it, named "Ridge test", you can run the script to test.
Re: Ridge penalty function question in Generalized regressionHi Eric,
One more suggestion for your customer to try is Partial Least Squares and in this case it is PLS-DA. The platform allows for variable selection, but uses all variables in the first pass of the modeling process. It is pretty fast over and could be useful for model comparison to see which modeling platform gives the best overall fit.
That said about PLS, I am with Clay on this one. I have used Gen Reg to analyze several spectral data sets and Elastic Net ranks high in building a good predictive model with data sets this wide.
Best regards,
Bill
Re: Ridge penalty function question in Generalized regressionThanks, @nick_shelton , You are correct, I try some data that variable with different scale and I found that once I standardize them before run ridge, it can save some time to get the result(In my test case, it can save 60% time spend ). Although it still take longer time than Elastic net and Lasso, I think it can save some time for those variables with different value range.
Re: Ridge penalty function question in Generalized regressionThanks, @Bill_Worley and @clay_barker, your suggestion and explanation are very helpful. I will try to convince my customer to try Elastic Net or PLS method for this case first.
Re: Ridge penalty function question in Generalized regressionThanks, Eric. I ran your 'Ridge Test' script, and it took 35,000 seconds (about 9.5 hours!). I suspect the machine you mentioned has a higher clock speed:
It was so poor I didn't bother to look further into the details (I typically use half of the RAM for a disk, and I didn't bother to change this).
Re: Ridge penalty function question in Generalized regressionThank you for your time and help, Ian.
Re: Ridge penalty function question in Generalized regressionHi @clay_barker, thank you for your reply, I have some questions need check with GR expert like you. Our customer want to know did you have any suggestion about "avoid " use Ridge as penalty method, when the analyze data over how many variables or observations? Moreover, just curious about do you suggest they to analyze those spectrum data by FDE? Thank you.
Re: Ridge penalty function question in Generalized regression@ian_jmp, here's the Tech Support Track, 7613489182. You might find what you're looking for in there.
CC: @AdamMorris, @clay_barker
