In the realm of machine learning and artificial intelligence, multiple linear regression (MLR) is a fundamental algorithm that has numerous applications in data analysis, prediction, and modeling. In this blog, we will delve into the math behind MLR, discuss its code implementation in Python, and explore how to create our own Multiple Linear Regression class.
Math Behind Multiple Linear Regression
Multiple Linear Regression is an extension of simple linear regression, where we have more than one independent variable (feature) to predict a continuous dependent variable. Mathematically, let’s consider a scenario with `n` observations, each described by:
- Independent variables: `X = [x1, x2, …, xn]`
- Dependent variable: `Y`
The goal is to find the best-fitting linear model that predicts `Y` based on the values of `X`.
Assuming a linear relationship between `Y` and each `Xi`, we can write:
`Y = β0 + β1 * X1 + β2 * X2 + … + βn * Xn`
where `β0` is the intercept, and `βi` are the coefficients for each feature `Xi`.
Ordinary Least Squares (OLS) Method
To estimate the model parameters, we use the Ordinary Least Squares (OLS) method. This involves minimizing the sum of the squared errors between observed responses and predicted values:
`minimize ∑(Y_i – (β0 + β1 * X1_i + … + βn * Xn_i))^2`
The OLS estimator for each parameter is given by:
`βi = (X^T * X)^-1 * X^T * Y`
where `X^T` is the transpose of the design matrix `X`, and `(X^T * X)^-1` is the inverse of the covariance matrix.
Python Code for Multiple Linear Regression
We’ll use Python with libraries like NumPy, pandas, and scikit-learn to implement multiple linear regression. Here’s an example code:
“`python
import numpy as np
from sklearn.linear_model import LinearRegression
import pandas as pd
# Generate sample data
np.random.seed(0)
X = np.random.rand(100, 3) # independent variables
y = 3 * X[:, 0] + 2 * X[:, 1] – 1 * X[:, 2] + np.random.randn(100)
# Create a design matrix (X) and target variable (y)
design_matrix = np.hstack((np.ones((100, 1)), X))
target_variable = y
# Train the model using LinearRegression
model = LinearRegression()
model.fit(design_matrix, target_variable)
# Get the coefficients
print(“Coefficients: “, model.coef_)
“`
In this example, we create a design matrix `X` with three independent variables and a target variable `y`. We then train a linear regression model using the scikit-learn library. The coefficients of the best-fitting line are printed to the console.
Creating Our Own Multiple Linear Regression Class
To gain more control over the MLR algorithm, we’ll create our own class that includes features like feature scaling, regularization, and cross-validation. Here’s an implementation:
“`python
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
class MultipleLinearRegression:
def __init__(self):
self.coef = None
self.intercept = None
self.score = None
self.r2 = None
def fit(self, X, y):
# Feature scaling using StandardScaler
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Regularization using Lasso
from sklearn.linear_model import Lasso
model = Lasso(alpha=0.1)
self.coef = model.fit(X_scaled, y).coef_
self.intercept = model.predict(np.array([np.ones((X.shape[0], 1))]).dot(self.coef))
def predict(self, X):
# Predict using the trained model
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.transform(X)
return self.intercept + np.dot(X_scaled, self.coef)
# Example usage:
np.random.seed(0)
X = np.random.rand(100, 3) # independent variables
y = 3 * X[:, 0] + 2 * X[:, 1] – 1 * X[:, 2] + np.random.randn(100)
model = MultipleLinearRegression()
model.fit(X, y)
print(“Coefficients: “, model.coef)
print(“Intercept: “, model.intercept)
“`
In this implementation, we create a `MultipleLinearRegression` class that includes the following features:
- Feature scaling using `StandardScaler`
- Regularization using `Lasso`, which reduces the effect of any single feature by shrinking its coefficients to zero
- Training and prediction methods
We can adjust the regularization parameter (`alpha`) to control the level of regularization. A small value (e.g., 0.1) will lead to some features being set to zero, while a large value (e.g., 100) will leave all coefficients unchanged.
Conclusion
In this blog, we explored multiple linear regression, including its math behind it, code implementation in Python using scikit-learn and our own class, and discussed the concept of regularization. We also touched upon some advanced topics such as cross-validation, hyperparameter tuning, and model selection.