Lasso

Lasso Regression Modeling

Lasso regression model modifies the linear regression equation by adding a penalty term to the sum of squared residuals (SSR) objective function, similar to ridge regression. Unlike ridge regression, which uses the L2 norm of the coefficients for regularization, lasso regression uses the L1 norm of the coefficients. The objective function of lasso regression can be written as Objective function = SSR + α * Σ(|βi|) where:

• SSR is the sum of squared residuals, which measures the discrepancy between the observed values of the dependent variable and the predicted values by the regression model.

• α is the regularization parameter, also known as the hyperparameter, that controls the strength of the penalty term. A higher value of α results in a stronger penalty, and a lower value of α results in a weaker penalty. It is a tuning parameter that we determine.

• Σ(|βi|) is the sum of the absolute values of the coefficients of the regression model, also known as the L1 norm of the coefficients. It represents the magnitude of the coefficients, and the penalty term is proportional to the absolute value of the magnitude of the coefficients.

The addition of the penalty term in the objective function of lasso regression results in a different estimation approach compared to ordinary least squares (OLS) used in linear regression. The penalty term induces sparsity in the estimated coefficients, meaning that it encourages some of the coefficients to be exactly equal to zero. This allows for automatic feature selection, where less important variables are effectively excluded from the model. Lasso regression is particularly useful when working with data sets that have a large number of predictors, and when feature selection is desired to identify the most relevant variables for predicting the dependent variable. We use it for regularization purposes to improve the stability and performance of linear regression models.

Lasso regression has the regression modeling ability to perform variable selection and regularization simultaneously. We use it for prediction, estimation, feature selection, and model regularization, where sparse models with a reduced number of predictors are desirable.