In the field of machine learning, data with many variables is often handled, and multicollinearity can occur when handling such data. In such cases, regularized regression such as Lasso, which is a method to avoid multicollinearity, is sometimes used.
In JMP Pro, regularized regression can be used as a method called "generalized regression," which enables stable model estimation that reduces the effects of multicollinearity by shrinking parameter estimates (regression coefficients) and selecting variables.
The following figure shows how the regression coefficients are shrunk by the Lasso. In the report "Estimates for Original Explanatory Variables," we can see that explanatory variables with small effects have been shrunk to 0, confirming that the Lasso is performing variable selection.
So how effective is Lasso regression for data with multicollinearity? In this article, we use hypothetical data that has a predetermined true relationship with the response (Y) to examine how well Lasso regression, multiple regression analysis, and stepwise methods can estimate the true model.
table of contents
Virtual Data
Response variable: Y
Explanatory variables: X1 to X20 (20 variables)
Y is generated from normal random numbers with the mean being a linear combination of several explanatory variables and a standard error of 10.
We will prepare two types of data in this format and compare the effectiveness of Lasso regression.
1. Data with little correlation between explanatory variables
2. Data with some degree of correlation between explanatory variables (multicollinearity)
This allows us to examine how the presence or absence of multicollinearity affects variable selection and model estimation .
*These two types of sample data are attached to this article.
1. Example of data with little correlation between explanatory variables
Data : "data_without_correlation.jmp"
Correlation between explanatory variables (r) : -0.06 to 0.10
Using this data, we fitted models using multiple regression analysis, stepwise regression (stopping rule: minimum BIC), and Lasso regression (validation method: BIC), and the regression coefficients obtained are summarized in the following table. The column "True Equation" lists the true coefficients of the explanatory variables used to generate Y.
In the stepwise method, the coefficients of variables not selected are displayed as 0, and in the Lasso regression, if a coefficient is shrunk to 0 by regularization, it is also displayed as 0.
These results confirm that both methods estimate models that are close to the true equation, with coefficients for ineffective variables being zero or very small.
In other words, in data with almost no correlation between explanatory variables, multiple regression analysis, stepwise methods, and Lasso regression were all able to estimate the true response relationship with broad accuracy.
2. Example of data with some degree of correlation between explanatory variables
Data : "data_without_correlation.jmp"
Correlation between explanatory variables (r)：-0.50～0.52
In such data, the correlation between explanatory variables makes the estimate of the regression coefficients unstable. If you run a multiple regression analysis on this data, the VIF will be very large, and the effects of multicollinearity will be evident.
As in the case of 1, we fitted models using multiple regression analysis, stepwise regression, and Lasso regression, and present a table summarizing the obtained coefficients.
In the multiple regression analysis, the effects of multicollinearity resulted in coefficients that were significantly different from the true values.
Even with the stepwise method, important variables were not selected, unnecessary variables were selected, and the correct variable selection based on the true equation was not performed.
On the other hand , with Lasso regression, as in case 1, we were able to shrink ineffective explanatory variables to 0 and obtain almost accurate estimates for effective variables.
3. Simulation for Lasso coefficients
In the examples so far, the Y values have been generated using the Random Normal function from a set of normal random numbers. However, since these are random numbers, the results may vary depending on the random numbers extracted.
Using the "Simulation" feature of JMP, we can repeat estimation based on a large number of sets of normal random numbers to obtain a simulated distribution of regression coefficients, which allows us to evaluate the validity and reliability of variable selection using Lasso regression.
Here, we use simulations to examine how frequently ineffective variables are shrunk to 0, and how frequently effective variables are estimated to be non-zero, when performing Lasso regression on "somewhat correlated data" (Case 2).
Steps for running a simulation in JMP
When running a Lasso regression, we show how to run 500 simulations on the estimates.
- Right-click on the "Estimated Value" number in the Lasso report "Estimates for Original Explanatory Variables" and select "Simulate."
- A simulation window will appear, so set the source column to Y and the destination column to Y. Set the sample size to 500 (the number of simulations), and the random seed value to 123456 (a value specified for convenience in order to reproduce the results).
This will output a new data table with the distribution of regression coefficients obtained from the 500 simulations.
If you run the script "Univariate distribution" on the output data table, the simulated distribution of each variable will be displayed. In addition, the "Number of zeros" and "Ratio of zeros" will be displayed in the "Summary statistics" report.
The number of zeros and the percentage of zeros for each variable obtained above are summarized in a table.
Variables with no effect in the true equation are shrunk to 0 at a high rate by Lasso regression, whereas variables with large absolute values of coefficients in the true equation (e.g., X7, X12) are never shrunk to 0 in any of the 500 simulations, indicating that they are always included in the model.
Simulations confirmed that Lasso regression can eliminate ineffective variables with a high probability and stably select important variables, even in cases of multicollinear data .
However, in practice, please note that in data with strong multicollinearity, Lasso regression has a tendency to select only one highly correlated explanatory variable and exclude the others.
by Naohiro Masukawa (JMP Japan)
Naohiro Masukawa - JMP User Community
