Introduction
Regularization techniques have become essential tools in machine learning, helping to prevent overfitting and improve model performance. Among these techniques, Lasso Regression stands out as a powerful method for regularizing linear models. In this blog post, we will delve into the math behind Lasso Regression, provide code examples in Python, and discuss the significance of this technique.
Math Behind Lasso Regression
Lasso Regression is a variant of Ridge Regression that uses L1 regularization instead of L2 regularization. The goal of Lasso Regression is to minimize the mean squared error (MSE) between the predicted values and actual values while regularizing the model coefficients. Mathematically, this can be represented as:
`minimize(0.5 * ||y – Xw||^2 + λ * |w|)`
where `y` is the response vector, `X` is the design matrix, `w` are the model coefficients, `λ` is the regularization parameter, and `||.||` denotes the L1 norm.
The key difference between Lasso Regression and Ridge Regression lies in the term `(λ * |w|)`. This term encourages the coefficients to be zero when λ is greater than 0. This leads to a sparsity in the model, where only non-zero coefficients are retained.
Code Implementation
We will implement Lasso Regression using Python’s scikit-learn library. First, let’s import the necessary libraries and load the iris dataset:
“`python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import Lasso, LinearRegression
“`
Next, we’ll create a simple function to implement Lasso Regression:
“`python
def lasso_regression(X, y, lambda_val):
model = Lasso(alpha=lambda_val, max_iter=10000)
model.fit(X, y)
return model.coef_
“`
We can use this function to fit the Lasso Regression model to our dataset and print the coefficients:
“`python
iris = load_iris()
X = iris.data
y = iris.target
# Create a linear regression object with lambda=0.1
model = LinearRegression()
model.fit(X, y)
# Print the coefficients
print(model.coef_)
# Create a lasso regression object with lambda=0.1
lasso_model = Lasso(alpha=0.1)
lasso_model.fit(X, y)
# Print the coefficients
print(lasso_model.coef_)
“`
Creating Our Own Lasso Regression Class
Now that we have a basic understanding of Lasso Regression, let’s create our own class from scratch:
“`python
import numpy as np
class LassoRegression:
def __init__(self, lambda_val=0.1):
self.lambda_val = lambda_val
self.coef_ = None
self.model = None
def _l2_norm(self, w):
return np.sqrt(np.sum(w**2))
def _l1_norm(self, w):
return np.sum(np.abs(w))
def fit(self, X, y):
n_samples, n_features = X.shape
theta = np.zeros(n_features)
# Gradient Descent for Lasso Regression
for i in range(10000):
# Compute the gradient of the objective function
gradient = -2 * X.T @ self.lambda_val * (X @ theta – y) / n_samples
# Update theta using gradient descent
theta -= gradient
# Store the updated coefficients
self.coef_ = np.copy(theta)
return self
def predict(self, X):
if self.model is None:
self.model = LinearRegression()
self.model.fit(X, np.zeros(X.shape[0]))
return self.model.predict(X)
# Usage example:
lasso_model = LassoRegression(lambda_val=0.1)
X_train = np.array([[1, 6], [2, 5], [3, 4]])
y_train = np.array([5, 4, 3])
lasso_model.fit(X_train, y_train)
print(lasso_model.coef_)
“`
Significance of Lasso Regression
Lasso Regression has several significance in machine learning:
1. Sparsity: The key advantage of Lasso Regression is its ability to produce sparse solutions, meaning that only non-zero coefficients are retained. This can be particularly useful when the number of features exceeds the number of samples.
2. Handling high-dimensional data: Lasso Regression is suitable for handling high-dimensional data where the number of features exceeds the number of samples.
3. Robustness to outliers: Since the L1 norm is used, Lasso Regression is more robust to outliers compared to Ridge Regression.
Conclusion
In this blog post, we explored the math behind Lasso Regression and implemented it using Python. We also created our own Lasso Regression class from scratch. By understanding how to use Lasso Regression effectively, we can leverage its benefits in machine learning, such as handling high-dimensional data, producing sparse solutions, and being more robust to outliers.
As with any regularization technique, the choice of λ (regularization parameter) is crucial for optimal performance. A small value of λ will produce a less sparse solution, while larger values will result in a sparser solution.
By incorporating Lasso Regression into your machine learning toolkit, you can tackle complex problems and make more informed decisions about your models.