nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Ridge penalty function question in Generalized regression

Eric_Wang

Our customer have some questions about the Generalized regression, according to their experience before, they want to use “Ridge” as the penalty function to construct the model because they don’t want to miss any variable’s effect, the “Elastic Net” or “Lasso” penalty function are not fit in this case.

However, customer usually analyzed spectrum data with many variables(some data contain 4500 variables with 65 rows), it took long time and can’t get the result when e use “Ridge” as the penalty function.

I had feedbacked the question to JMP support, they try the sample I gave( as attachment, around 2400 variables and 65 observations ) on Windows 10 machine, with 16GB RAM and 4 cores. And JMP went into "Not Responding" mode for the first hour, but then produced a progress bar. It took long time to get the ridge result(4.5hr), even change the grid point to 50.

Compare with Elastic net or Lasso method, those run about 2~3 minutes, there is big time spend difference between Elastic net/Lasso and Ridge

Moreover, customer also mentioned that Python can execute the Generalized regression with “Ridge” as the penalty function more quickly.

Is it possible to speed up the model fit process? Something like making CPU full loading? Or other suitable feature/platform can help customer to get estimate result more efficiency?

I attached the sample data as well. Thank you

new ridge data.jmp

Accepted answers

All comments

ian_jmp

FWIW, I have a Windows 10 box with 32 cores and 128GB of RAM. I'm not sure what JMP will take advantage of, but I'd be happy to try to get another benchmark if it helps. But the details matter (particularly if it takes five hours for a run . . . ). So could you post the script to do the ridge regression, please? I didn't see it in the table.

nick_shelton

@Eric_Wang Some implementations of Ridge Regression in Python require input variables to be standardized. Standardizing the variables could speed up the time to solve for the coefficients. I'm not sure if JMP's routine automatically performs some kind of standardization for the inputs but you can experiment if standardizing the ~4500 input variables allows for a quicker compute time.

Jeff_Perkinson

@ian_jmp, here's the Tech Support Track, 7613489182. You might find what you're looking for in there.

CC: @AdamMorris, @clay_barker

clay_barker

This sounds like a case where we should be using the elastic net. In cases like this where there are far more predictors than observations, the big concern when doing variable selection is that the collinearity and redundancy will cause us to miss predictors that truly matter. If X1 and X2 are highly correlated, something like forward selection is likely to only pick one of them. The strength of the elastic net is that it will select groups of correlated predictors, so X1 and X2 in this case. My guess is that in this situation "don’t want to miss any variable’s effect" actually means that they do not want to miss any effects that might have a real impact. The elastic net is built for this exact scenario.

So why are lasso and elastic so much faster than ridge regression here? Because they do variable selection, elastic net and lasso computations only involve a handful of predictors while ridge regression is doing optimization over all 2400 variables specified. It's also important to note that lasso and elastic net both tend to dominate ridge regression in terms of fit quality and interpretation, so we've made efforts to improve computation times for lasso and elastic net fitters. We have not been able to prioritize similar improvements for ridge regression.

Eric_Wang

Thanks, Ian, I modified and reattached the sample data with the script in script area of it, named "Ridge test", you can run the script to test.

Bill_Worley

Hi Eric,

One more suggestion for your customer to try is Partial Least Squares and in this case it is PLS-DA. The platform allows for variable selection, but uses all variables in the first pass of the modeling process. It is pretty fast over and could be useful for model comparison to see which modeling platform gives the best overall fit.

That said about PLS, I am with Clay on this one. I have used Gen Reg to analyze several spectral data sets and Elastic Net ranks high in building a good predictive model with data sets this wide.

Best regards,

Bill

Eric_Wang

Thanks, @nick_shelton , You are correct, I try some data that variable with different scale and I found that once I standardize them before run ridge, it can save some time to get the result(In my test case, it can save 60% time spend ). Although it still take longer time than Elastic net and Lasso, I think it can save some time for those variables with different value range.

Eric_Wang

Thanks, @Bill_Worley and @clay_barker, your suggestion and explanation are very helpful. I will try to convince my customer to try Elastic Net or PLS method for this case first.

ian_jmp

Thanks, Eric. I ran your 'Ridge Test' script, and it took 35,000 seconds (about 9.5 hours!). I suspect the machine you mentioned has a higher clock speed:

Screenshot 2022-01-25 at 15.24.17.png

It was so poor I didn't bother to look further into the details (I typically use half of the RAM for a disk, and I didn't bother to change this).

Eric Ridge Regression.jrn

Eric_Wang

Thank you for your time and help, Ian.

Eric_Wang

Hi @clay_barker, thank you for your reply, I have some questions need check with GR expert like you. Our customer want to know did you have any suggestion about "avoid " use Ridge as penalty method, when the analyze data over how many variables or observations? Moreover, just curious about do you suggest they to analyze those spectrum data by FDE? Thank you.