Blog

Multiple linear regression in machine learning is a supervised algorithm that models the relationship between a dependent variable and multiple independent variables. This relationship is used to predict the outcome of the dependent variable.

Multiple linear regression is a type of linear regression in machine learning. There are mainly two types of linear regression algorithms −

• simple linear regression − it deals with two features (one dependent variable and one independent variable).
• multiple linear regression − deals with more than two features (one dependent variable and more than one independent variables).

Let’s discuss multiple linear regression in detail −

What is Multiple Linear Regression?

In machine learning, multiple linear regression (MLR) is a statistical technique that is used to predict the outcome of a dependent variable based on the values of multiple independent variables. The multiple linear regression algorithm is trained on data to learn a relationship (known as a regression line) that best fits the data. This relation describes how various factors affect the result. This relation is used to forecast the value of dependent variable based on the values of independent variables.

In linear regression (simple and multiple), the dependent variable is continuous (numeric value) and independent variables can be continuous or discreet (numeric value). Independent variables can also be categorical (gender, occupation), but they need to be converted to numerical values first.

Multiple linear regression is basically the extension of simple linear regression that predicts a response using two or more features. Mathematically we can represent the multiple linear regression as follows −

Consider a dataset having n observations, p features i.e. independent variables and y as one response i.e. dependent variable the regression line for p features can be calculated as follows −

h(xi)=w0+w1xi1+w2xi2+⋅⋅⋅+wpxip

Here, h(xi) is the predicted response value and w0,w1,w2….wp are the regression coefficients.

Multiple Linear Regression models always includes the errors in the data known as residual error which changes the calculation as follows −

yi=w0+w1xi1+w2xi2+⋅⋅⋅+wpxip+ei

We can also write the above equation as follows −

yi=h(xi)+eiorei=yi−h(xi)

Assumptions of Multiple Linear Regression

The following are some assumptions about the dataset that are made by the multiple linear regression model −

1. Linearity

The relationship between the dependent variable (target) and independent (predictor) variables is linear.

2. Independence

Each observation is independent of others. The value of the dependent variable for one observation is independent of the value of another.

3. Homoscedasticity

For all observations, the variance of the residual errors is similar across the value of each independent variable.

4. Normality of Errors

The residuals (errors) are normally distributed. The residuals are differences between the actual and predicted values.

5. No Multicollinearity

The independent variables are not highly correlated with each other. Linear regression models assume that there is very little or no multi-collinearity in the data.

6. No Autocorrelation

There is no correlation between residuals. This ensures that the residuals (errors) are independent of each other.

7. Fixed Independent Variables

The values of independent variables are fixed in all repeated samples.

Violations of these assumptions can lead to biased or inefficient estimates. It is essential to validate these assumptions to ensure model accuracy.

Implementing Multiple Linear Regression in Python

To implement multiple linear regression in Python using Scikit-Learn, we can use the same LinearRegression class as in simple linear regression, but this time we need to provide multiple independent variables as input.

Step 1: Data Preparation

We use the dataset named data.csv with 50 examples. It contains four predictor (independent) variables and a target (dependent) variable. The following table represents the data in data.csv file.

data.csv

R&D Spend	Administration	Marketing Spend	State	Profit
165349.2	136897.8	471784.1	New York	192261.8
162597.7	151377.6	443898.5	California	191792.1
153441.5	101145.6	407934.5	Florida	191050.4
144372.4	118671.9	383199.6	New York	182902
142107.3	91391.77	366168.4	Florida	166187.9
131876.9	99814.71	362861.4	New York	156991.1
134615.5	147198.9	127716.8	California	156122.5
130298.1	145530.1	323876.7	Florida	155752.6
120542.5	148719	311613.3	New York	152211.8
123334.9	108679.2	304981.6	California	149760
101913.1	110594.1	229161	Florida	146122
100672	91790.61	249744.6	California	144259.4
93863.75	127320.4	249839.4	Florida	141585.5
91992.39	135495.1	252664.9	California	134307.4
119943.2	156547.4	256512.9	Florida	132602.7
114523.6	122616.8	261776.2	New York	129917
78013.11	121597.6	264346.1	California	126992.9
94657.16	145077.6	282574.3	New York	125370.4
91749.16	114175.8	294919.6	Florida	124266.9
86419.7	153514.1	0	New York	122776.9
76253.86	113867.3	298664.5	California	118474
78389.47	153773.4	299737.3	New York	111313
73994.56	122782.8	303319.3	Florida	110352.3
67532.53	105751	304768.7	Florida	108734
77044.01	99281.34	140574.8	New York	108552
64664.71	139553.2	137962.6	California	107404.3
75328.87	144136	134050.1	Florida	105733.5
72107.6	127864.6	353183.8	New York	105008.3
66051.52	182645.6	118148.2	Florida	103282.4
65605.48	153032.1	107138.4	New York	101004.6
61994.48	115641.3	91131.24	Florida	99937.59
61136.38	152701.9	88218.23	New York	97483.56
63408.86	129219.6	46085.25	California	97427.84
55493.95	103057.5	214634.8	Florida	96778.92
46426.07	157693.9	210797.7	California	96712.8
46014.02	85047.44	205517.6	New York	96479.51
28663.76	127056.2	201126.8	Florida	90708.19
44069.95	51283.14	197029.4	California	89949.14
20229.59	65947.93	185265.1	New York	81229.06
38558.51	82982.09	174999.3	California	81005.76
28754.33	118546.1	172795.7	California	78239.91
27892.92	84710.77	164470.7	Florida	77798.83
23640.93	96189.63	148001.1	California	71498.49
15505.73	127382.3	35534.17	New York	69758.98
22177.74	154806.1	28334.72	California	65200.33
1000.23	124153	1903.93	New York	64926.08
1315.46	115816.2	297114.5	Florida	49490.75
0	135426.9	0	California	42559.73
542.05	51743.15	0	New York	35673.41
0	116983.8	45173.06	California	14681.4

You can create a CSV file and store the above data points in it.

We have our dataset as data.csv file. We will use it to understand the implementation of the multiple linear regression in Python.

We need to import libraries before loading the dataset.

# import librariesimport numpy as np
import matplotlib.pyplot as plt
import pandas as pd

Load the dataset

We load our dataset as a Pandas Data frame named <string>dataset. Now let’s create a list of independent values (predictors) and put them in a variable called X.</string>

The independent values are ‘R&D Spend’, ‘Administration’, ‘Marketing Spend’. We are not using the independent variable ‘State’ for sake of simplicity.

We put the dependent variable values to a variable y.

# load dataset
dataset = pd.read_csv('data.csv')
X = dataset[['R&D Spend','Administration','Marketing Spend']]
y = dataset['Profit']

Let’s check first five examples (rows) of input features (X) and target (y) −

X.head()

Output

	R&D Spend	Administration	Marketing Spend
0	165349.20	136897.80	471784.10
1	162597.70	151377.59	443898.53
2	153441.51	101145.55	407934.54
3	144372.41	118671.85	383199.62
4	142107.34	91391.77	366168.42

y.head()

Output

	Profit
0	192261.83
1	191792.06
2	191050.39
3	182901.99
4	166187.94

Split the dataset into training and test sets

Now, we split the dataset into a training set and a test set. Both the X(independent values) and y (dependent values) are divided into two sets – training and test. We will use 20% for the test set. In such a way out of 50 feature vectors (observations/ examples), there will be 40 feature vectors in training set and 10 feature vectors in test set.

# Split the dataset into training and test sets from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size =0.2)

Here X_train and X_test represent input features in training set and test set, where y_train and y_test represent target values (output) in traning and test set.

Step 2: Model Training

The next step is to fit our model with training data. We will use linear_model class from sklearn module. We use the Linear Regression() method of linear_model class to create a linear regression object, here we name it as regressor.

# Fit Multiple Linear Regression to the Training setfrom sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

The regressor object has fit() method. The fit() method is used to fit the linear regression object, regressor to the training data. The model learns the relation between the predictor variable (X_train), and the target variable (y_train).

Step 3: Model Testing

Now our model is ready to use for prediction. Let’s test our regressor model on test data.

We use predict() method to predict the results for the test set. It takes input features (X_test) and return the redicted values.

y_pred = regressor.predict(X_test)
df = pd.DataFrame({'Real Values':y_test,'Predicted Values':y_pred})print(df)

Output

	Real Values	Predicted Values
23	108733.99	110159.827849
43	69758.98	59787.885207
26	105733.54	110545.686823
34	96712.80	88204.710014
24	108552.04	114094.816702
39	81005.76	84152.640761
44	65200.33	63862.256006
18	124266.90	129379.514419
47	42559.73	45832.902722
17	125370.37	130086.829016

You can compare the actual values and predicted values.

Step 4: Model Evaluation

We now evaluate our model to check how accurate it is. We will use mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), and R²-score (Coefficient of determination).

from sklearn.metrics import mean_squared_error, root_mean_squared_error, mean_absolute_error, r2_score
# Assuming you have your true y values (y_test) and predicted y values (y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = root_mean_squared_error(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)print("Mean Squared Error (MSE):", mse)print("Root Mean Squared Error (RMSE):", rmse)print("Mean Absolute Error (MAE):", mae)print("R-squared (R2):", r2)

Output

Mean Squared Error (MSE): 72684687.6336162
Root Mean Squared Error (RMSE): 8525.531516193943
Mean Absolute Error (MAE): 6425.118502810154
R-squared (R2): 0.9588459519573707

You can examine the above metrics. Our model shows an R-squared score of around 0.96, which means that 96% of data points are scattered around the fitted regression line. Another interpretation is that 96% of the variation in the output variables is explained by the input variables.

Step 5: Model Prediction for New Data

Let’s use our regressor model to predict profit values based on R&D Spend, Administration and Marketing Spend.

[‘R&D Spend’,’Administration’,’Marketing Spend’]=[166343.2, 136787.8, 461724.1]

// predict profit when R&D Spend is166343.2, Administration is136787.8and Marketing Spend is461724.1
new_data =[[166343.2,136787.8,461724.1]] 
profit = regressor.predict(new_data)print(profit)

Output

[193053.61874652]

The model predicts the profit value is approximately 192090.567 for the above three values.

Model Parameters (Coefficients and Intercept)

The model parameters (intercept and coefficients) describe the relation between a dependent variable and the independent variables.

Our regression model for the above use case,

Y=w0+w1X1+w2X2+w2X2

w0 is intercept and w1,w2,w3 are coefficients of X1,X2,X3 respectively.

Here,

• X1 represents R&D Spend,
• X2 represents Administration, and
• X3 represents Marketing Spend.

Let’s first compute the intercept and coefficients.

print("coefficients: ", regressor.coef_)print("intercept: ", regressor.intercept_)

Output

coefficients: [ 0.81129358 -0.06184074  0.02515044]
intercept: 54946.94052163202

The above output shows the following –

• w0 = 54946.94052163202
• w1 = 0.81129358
• w2 = -0.06184074
• w3 = 0.02515044

Result Explanation

We have calculated intercept (w0) and coefficients (w1, w2, w3).

The coefficients are as follows –

• R&D Spend: 0.81129358
• Administration: -0.06184074
• Marketing Spend: 0.02515044

This shows that if R&D Spend is increased by 1 USD, the Profit will increase by 0.81851334 USD.

The result shows that when Administration spend is increased by 1 USD, the Profit will decrease by 0.03124763 USD.

And when Marketing Spend increases by 1 USD, the Profit increases by 0.02042286 USD.

Let’s verify the result,

In step 5, we have predicted Profit for new data as 193053.61874652

Here,

new_data =[[166343.2,136787.8,461724.1]] 
Profit =54946.94052163202+0.81129358*166343.2-0.06184074*136787.8+0.02515044*461724.1
Profit =193053.616257

Which is approximately the same as model prediction. Why approximately? Because of residual error.

residual error = 193053.61874652 - 193053.616257
residual error = 0.00248952

Applications of Multiple Linear Regression

The following are some commonly used applications of multiple linear regression −

Application	Description
Finance	Predicting stock prices, forecasting exchange rates, assessing credit risk.
Marketing	Predicting sales, customer churn, and marketing campaign effectiveness.
Real Estate	Predicting house prices based on factors like size, location, and number of bedrooms.
Healthcare	Predicting patient outcomes, analyzing the impact of treatments, and identifying risk factors for diseases.
Economics	Forecasting economic growth, analyzing the impact of policies, and predicting inflation rates.
Social Sciences	Modeling social phenomena, predicting election outcomes, and understanding human behavior.

Challenges of Multiple Linear Regression

The following are some common challenges faced by multiple linear regression in machine learning −

Challenge	Description
Multicollinearity	High correlation between independent variables, leading to unstable model coefficients and difficulty in interpreting the impact of individual variables.
Overfitting	The model fits the training data too closely, leading to poor performance on new, unseen data.
Underfitting	The model fails to capture the underlying patterns in the data, resulting in poor performance on both training and test data.
Non-linearity	Multiple linear regression assumes a linear relationship between the independent and dependent variables. Non-linear relationships can lead to inaccurate predictions.
Outliers	Outliers can significantly impact the model’s performance, especially in small datasets.
Missing Data	Missing data can lead to biased and inaccurate results.

Difference Between Simple and Multiple Linear Regression

The following table highlights the major differences between simple and multiple linear regression −

Feature	Simple Linear Regression	Multiple Linear Regression
Independent Variables	One	Two or more
Model Equation	y = w1x + w0	y=w0+w1x1+w2x2+ … +wpxp
Complexity	Less complex	More complex due to multiple variables
Real-world Applications	Predicting house prices based on square footage, predicting sales based on advertising expenditure	Predicting sales based on advertising expenditure, price, and competitor activity, predicting student performance based on study hours, attendance, and IQ
Model Interpretation	Easier to interpret coefficients	More complex to interpret due to multiple variables

What is Simple Linear Regression?

Simple linear regression is a statistical and supervised learning method in which a single independent variable (also known as a predictor variable) is used to predict the dependent variable. In other words, it models the linear relationship between the dependent variable and a single independent variable.

Simple linear regression in machine learning is a type of linear regression. When the linear regression algorithm deals with a single independent variable, it is known as simple linear regression. When there is more than one independent variable (feature variables), it is known as multiple linear regression.

Independent Variable

The feature inputs in the dataset are termed as the independent variables. There is only a single independent variable in simple linear regression. An independent variable is also known as a predictor variable as it is used to predict the target value. It is plotted on a horizontal axis.

Dependent Variable

The target value in the dataset is termed as the dependent variable. It is also known as a response variable or predicted variable. It is plotted on a vertical axis.

Line of Regression

In simple linear regression, a line of regression is a straight line that best fits the data points and is used to show the relationship between a dependent variable and an independent variable.

Graphical Representation

The following graph depicts the simple linear regression model −

In the above image, the straight line represents the simple linear regression line where &Ycirc; is the predicted value, and Y is dependent variable (target) and X is independent variable (input).

Simple Linear Regression Model

A simple linear regression model in machine learning can be represented as the following mathematical equation −

Y=w0+w1X+ϵ

Where

• Y is the dependent variable (target).
• X is the independent variable (feature).
• w₀ is the y-intercept of the line.
• w₁ is the slope of the line, representing the effect of X on Y.
• ε is the error term, capturing the variability in Y not explained by X.

How Simple Linear Regression Works?

The main of simple linear regression is to find the best fit line (a straight line) through the data points that minimizes the difference between the actual values and predicted values.

Defining Hypothesis Function

In simple linear regression, the hypothesis is that there is a linear relation between the dependent variable (output/ target) and the independent variable (input). This linear relation can be represented using a linear equation −

Ŷ =w0+w1X

With different values of parameters w₀ and w₁ there are multiple linear equations (straight lines). The set of all such linear equations (all straight lines) is termed hypothesis space.

Now, the main aim of the simple linear regression model is to find the best-fit line in Hypothesis space (set of all straight lines).

Finding the Best Fit Line

Now the task is to find the best fit line (line of regression). To do this, we define a cost function or loss function that measure the the difference between the actual values and predicted values.

To find the best fit line, the simple linear regression model initializes (with default values) the parameters of the regression line. This regression line (with initialized parameters) is used to find the predicted values for the given input values.

Loss Function for Simple Linear Regression

Now using the input and predicted values, we compute the loss function. The loss function is used to find the optimal values of the parameters.

The loss function finds the difference between the input value and predicted value. There are different loss functions such as mean squared error (MSE), mean absolute error (MEA), R-squared, etc. used in simple linear regression. The most commonly used loss function is mean squared error.

The loss function for simple linear regression in terms of mean squared error is as follows −

J(w0,w1)=12n∑i=1n(Yi−Ŷ i)2

Optimization

The optimal values of parameters are those values that minimize the cost function. Finding the optimal values is an iterative process in which the parameters are updated iteratively.

There are many optimization techniques applied in simple linear regression. Gradient Descent is a simple and most common optimization technique used in simple linear regression.

A linear equation with optimal parameter values is the best fit line(regression line) and it is the final solution for a simple linear regression problem. This line is used to predict new and unseen data.

Assumptions of Simple Linear Regression

There are some assumptions about the dataset that are made by the simple linear regression model. The following are some assumptions −

• Linearity − This assumption assumes that the relationship between the dependent and independent variables is linear. That means the dependent variable changes linearly as the independent variable changes. A scatter plot will show the linearity in the dataset.
• Homoskedasticity − For all observations, the variance of the residuals is the same. This assumption relates to the squared residuals.
• Independence − The examples (observations or X and Y pairs) are independent. There is no collinearity in data so the residuals will not be correlated. To check this, we example the scatter plot of residuals vs. fits.
• Normality − Model Residuals are normally distributed. Residuals are the differences between the actual and predicted values. To check for the normality, we examine the histogram of residuals. The histogram should be approximately normally distributed.

Implementation of Simple Linear Regression Algorithm using Python

To implement the simple linear regression algorithm, we are taking a dataset with two variables: YearsExperience (independent variable) and Salary (dependent variable).

Here, we are using the following dataset. The dataset contains 30 examples of data points. You can create a CSV file and store these data points in it.

Salary_Data.csv

Years of Experience	Salary
1.1	39343
1.3	46205
1.5	37731
2	43525
2.2	39891
2.9	56642
3	60150
3.2	54445
3.2	64445
3.7	57189
3.9	63218
4	55794
4	56957
4.1	57081
4.5	61111
4.9	67938
5.1	66029
5.3	83088
5.9	81363
6	93940
6.8	91738
7.1	98273
7.9	101302
8.2	113812
8.7	109431
9	105582
9.5	116969
9.6	112635
10.3	122391
10.5	121872

What is the purpose of this implementation?

The purpose of building this simple linear regression model is to determine which line best represents the relationship between the two variables.

The following are the steps to implement the simple linear regression model in Python −

Step 1: Data Preparation

Data preparation or pre-processing is the initial step. We have our dataset as a CSV file named “Salary_Data.csv,” as discussed above.

We need to import python libraries prior to importing the dataset and building the simple linear regression model.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

Load the dataset

dataset = pd.read_csv('Salary_Data.csv')

The dependent variable (X) and independent variable (Y) must then be extracted from the provided dataset. Years of experience (YearsExperience) is the independent variable, and Salary is the dependent variable.

X = dataset.iloc[:,:-1].values
y = dataset.iloc[:,-1].values

Let’s check the first five examples of the dataset.

print(dataset.head())

Output

0	1.1	39343.0
1	1.3	46205.0
2	1.5	37731.0
3	2.0	43525.0
4	2.2	39891.0

Lets check if the dataset is linear or not

plt.scatter(X, y, color="green")
plt.title("Salary vs Experience")
plt.xlabel("Years of Experience")
plt.ylabel("Salary (INR)")
plt.show()

Output

The above graph shows that the dependent and independent variables are linearly dependent. So we can apply the simple linear regression on the dataset to find the best relation between these variables.

Split the dataset into training and testing sets

The training set and test set will then be divided into two groups. We will use 80% observations for the training set and 20% observations for the test set out of the total 30 observations we have. So there will be 24 observation in training set and 6 observation in test set. We divide our dataset into training and test sets so that we can use one set to train and the other to test our model.

# Split the dataset into training and testing setsfrom sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size =0.2)

Here, X_train represents the input feature of the training data and y_train represents the output variable (target variable).

Step 2: Model Training (Fitting the Simple Linear Regression to Training Set)

The next step is fitting our model with the training dataset. We will use scikit-learn’s LinearRegression class to train a simple linear regression model on the training data. The code for this is as follows −

from sklearn.linear_model import LinearRegression

# Create a linear regression object
regressor= LinearRegression()
regressor.fit(X_train, y_train)

The fit() method is used to fit the linear regression object (regressor) to the training data. The model learns the relation between the predictor variable (X_train), and the target variable (y_train).

Step 3: Model Testing

Once the model is trained, we can use it to make predictions on the test data. The code for this is as follows −

y_pred = regressor.predict(X_test)
df = pd.DataFrame({'Actual Values':y_test,'Predicted Values':y_pred})print(df)

Output

   Actual Values  Predicted Values
0        60150.0      54093.648425
1        93940.0      82416.119864
2        57081.0      64478.554619
3       116969.0     115459.003211
4        56957.0      63534.472238
5       121872.0     124899.827024

The above output shows actual values and predicted values of Salary in the test set.

Here, X_test represents the input feature of the test data and y_pred represents the predicted output variable (target variable).

Similarly, you can test the model with training data.

y_pred = regressor.predict(X_train)
df = pd.DataFrame({'Real Values':y_test,'Predicted Values':y_pred})print(df)

Output

    Real Values  Predicted Values
0       57189.0      60702.225094
1       64445.0      55981.813188
2       63218.0      62590.389857
3      122391.0     123011.662261
4       91738.0      89968.778915
5       43525.0      44652.824612
6       61111.0      68254.884145
7       56642.0      53149.566044
8       66029.0      73919.378433
9       83088.0      75807.543195
10      46205.0      38044.247943
11     109431.0     107906.344160
12      98273.0      92801.026059
13      37731.0      39932.412705
14      54445.0      55981.813188
15      39891.0      46540.989374
16     101302.0     100353.685109
17      55794.0      63534.472238
18      81363.0      81472.037483
19      39343.0      36156.083180
20     113812.0     103185.932253
21      67938.0      72031.213670
22     112635.0     116403.085592
23     105582.0     110738.591304

Step 4: Model Evaluation

We need to evaluate the performance of the model to determine its accuracy. We will use the mean squared error (MSE), root mse (RMSE), mean average error (MAE), and the coefficient of determination (R^2) as evaluation metrics. The code for this is as follows −

from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import r2_score

# get the predicted values for test dat
y_pred = regressor.predict(X_test)
mse = mean_squared_error(y_test, y_pred)print("mse", mse)
rmse = mean_squared_error(y_test, y_pred, squared=False)print("rsme", rmse)
mae = mean_absolute_error(y_test, y_pred)print("mae", mae)
r2 = r2_score(y_test, y_pred)print("r2", r2)

Output

mse:  46485664.99327367
rsme:  6818.0396737826095
mae:  6015.513730219523
r2:  0.9399326805390613

Here, y_test represents the actual output variable of the test data.

Step 5: Visualize Training Set Results (with Regression Line)

Now, let’s visualize the results on the training set and the regression line.

We use the scatter plot to plot the actual values (input and target values) in the training set. We also plot a straight line (regression line) for actual values (input) and predicted values of the training set.

y_pred = regressor.predict(X_train)
plt.scatter(X_train, y_train, color="green", label="training data points (actual)")
plt.scatter(X_train, y_pred, color="blue",label="training data points (predicted)")
plt.plot(X_train, y_pred, color="red")
plt.title("Salary vs Experience (Training Dataset)")
plt.xlabel("Years of Experience")
plt.ylabel("Salary(In Rupees)")
plt.legend()
plt.show()

Output

The above graph shows the line of regression (straight line in red color), actual values (in green color), and predicted values (in blue color) for the training set.

Step 6: Visualize the Test Set Results (with Regression Line)

Now, let’s visualize the results on the test set and the regression line.

We use the scatter plot to plot the actual values (input and target values) in the test set. We also plot a straight line (regression line) for actual values (input) and predicted values of the test set.

y_pred = regressor.predict(X_test)
plt.scatter(X_test, y_test, color="green", label="test data points (actual)")
plt.scatter(X_test, y_pred, color="blue",label="test data points (predicted)")
plt.plot(X_test, y_pred, color="red")
plt.title("Salary vs Experience (Test Dataset)")
plt.xlabel("Years of Experience")
plt.ylabel("Salary(In Rupees)")
plt.legend()
plt.show()

Output

The above graph shows the line of regression (straight line in red color), actual values (in green color), and predicted values (in blue color) for the test set.

Linear regression in machine learning is defined as a statistical model that analyzes the linear relationship between a dependent variable and a given set of independent variables. The linear relationship between variables means that when the value of one or more independent variables will change (increase or decrease), the value of the dependent variable will also change accordingly (increase or decrease).

In machine learning, linear regression is used for predicting continuous numeric values based on learned linear relation for new and unseen data. It is used in predictive modeling, financial forecasting, risk assessment, etc.

In this chapter, we will discuss the following topics in detail −

• What is Linear Regression?
• Types of Linear Regression
• How Does Linear Regression Work?
• Hypothesis Function For Linear Regression
• Finding the Best Fit Line
• Loss Function For Linear Regression
• Gradient Descent for Optimization
• Assumptions of Linear Regression
• Evaluation Metrics for Linear Regression
• Applications of Linear Regression
• Advantages of Linear Regression
• Common Challenges with Linear Regression

What is Linear Regression?

Linear regression is a statistical technique that estimates the linear relationship between a dependent and one or more independent variables. In machine learning, linear regression is implemented as a supervised learning approach. In machine learning, labeled datasets contain input data (features) and output labels (target values). For linear regression in machine learning, we represent features as independent variables and target values as the dependent variable.

For the simplicity, take the following data (Single feature and single target)

Square Feet (X)	House Price (Y)
1300	240
1500	320
1700	330
1830	295
1550	256
2350	409
1450	319

In the above data, the target House Price is the dependent variable represented by X, and the feature, Square Feet, is the independent variable represented by Y. The input features (X) are used to predict the target label (Y). So, the independent variables are also known as predictor variables, and the dependent variable is known as the response variable.

So lets define linear regression in machine learning as follows:

In machine learning, linear regression uses a linear equation to model the relationship between a dependent variable (Y) and one or more independent variables (Y).

The main goal of the linear regression model is to find the best-fitting straight line (often called a regression line) through a set of data points.

Line of Regression

A straight line that shows a relation between the dependent variable and independent variables is known as the line of regression or regression line.

Furthermore, the linear relationship can be positive or negative in nature as explained below −

1. Positive Linear Relationship

A linear relationship will be called positive if both independent and dependent variable increases. It can be understood with the help of the following graph −

2. Negative Linear Relationship

A linear relationship will be called positive if the independent increases and the dependent variable decreases. It can be understood with the help of the following graph −

Linear regression is of two types, “simple linear regression” and “multiple linear regression”, which we are going to discuss in the next two chapters of this tutorial.

Types of Linear Regression

Linear regression is of the following two types −

• Simple Linear Regression
• Multiple Linear Regression

1. Simple Linear Regression

Simple linear regression is a type of regression analysis in which a single independent variable (also known as a predictor variable) is used to predict the dependent variable. In other words, it models the linear relationship between the dependent variable and a single independent variable.

In the above image, the straight line represents the simple linear regression line where &Ycirc; is the predicted value, and X is the input value.

Mathematically, the relationship can be modeled as a linear equation −

Y=w0+w1X+ϵ

Where

• Y is the dependent variable (target).
• X is the independent variable (feature).
• w₀ is the y-intercept of the line.
• w₁ is the slope of the line, representing the effect of X on Y.
• ε is the error term, capturing the variability in Y not explained by X.

2. Multiple Linear Regression

Multiple linear regression is basically the extension of simple linear regression that predicts a response using two or more features.

When dealing with more than one independent variable, we extend simple linear regression to multiple linear regression. The model is expressed as:

Multiple linear regression extends the concept of simple linear regression to multiple independent variables. The model is expressed as:

Y=w0+w1X1+w2X2+⋯+wpXp+ϵ

Where

• X₁, X₂, …, X_p are the independent variables (features).
• w₀, w₁, …, w_p are the coefficients for these variables.
• ε is the error term.

How Does Linear Regression Work?

The main goal of linear regression is to find the best-fit line through a set of data points that minimizes the difference between the actual values and predicted values. So it is done? This is done by estimating the parameters w₀, w₁ etc.

The working of linear regression in machine learning can be broken down into many steps as follows −

• Hypothesis − We assume that there is a linear relation between input and output.
• Cost Function − Define a loss or cost function. The cost function quantifies the model’s prediction error. The cost function takes the model’s predicted values and actual values and returns a single scaler value that represents the cost of the model’s prediction.
• Optimization − Optimize (minimize) the model’s cost function by updating the model’s parameters.

It continues updating the model’s parameters until the cost or error of the model’s prediction is optimized (minimized).

Let’s discuss the above three steps in more detail −

Hypothesis Function For Linear Regression

In linear regression problems, we assume that there is a linear relationship between input features (X) and predicted value (&Ycirc;).

The hypothesis function returns the predicted value for a given input value. Generally we represent a hypothesis by h_w(X) and it is equal to &Ycirc;.

Hypothesis function for simple linear regression −

Ŷ =w0+w1X

Hypothesis function for multiple linear regression −

Ŷ =w0+w1X1+w2X2+⋯+wpXp

For different values of parameters (weights), we can find many regression lines. The main goal is to find the best-fit lines. Let’s discuss it as below −

Finding the Best Fit Line

We discussed above that different set of parameters will provide different regression lines. However, each regression line does not represent the optimal relation between the input and output values. The main goal is to find the best-fit line.

A regression line is said to be the best fit if the error between actual and predicted values is minimal.

Below image shows a regression line with error (ε) at input data point X. The error is calculated for all data points and our goal is to minimize the average error/ loss. We can use different types of loss functions such as mean square error (MSE), mean average error (MAE), L₁ loss, L₂ Loss, etc.

So, how can we minimize the error between the actual and predicted values? Let’s discuss the important concept, which is cost function or loss function.

Loss Function for Linear Regression

The error between actual and predicted values can be quantified using a loss function of the cost function. The cost function takes the model’s predicted values and actual values and returns a single scaler value that represents the cost of the model’s prediction. Our main goal is to minimize the cost function.

The most commonly used cost function is the mean squared error function.

J(w0,w1)=12n∑i=1n(Yi−Ŷ i)2

Where,

• n is the number of data points.
• Y_i is the observed value for the i-th data point.
• Ŷ i=w0+w1Xi is the predicted value for the i-th data point.

Gradient Descent for Optimization

Now we have defined our loss function. The next step is to minimize it and find the optimized values of the parameters or weights. This process of finding optimal values of parameters such that the loss or error is minimal is known as model optimization.

Gradient Descent is one of the most used optimization techniques for linear regression.

To find the optimal values of parameters, gradient descent is often used, especially in cases with large datasets. Gradient descent iteratively adjusts the parameters in the direction of the steepest descent of the cost function.

The parameter updates are given by

w0=w0−α∂J∂w0

w1=w1−α∂J∂w1

Where α is the learning rate, and the partial derivatives are:

∂J∂w0=−1n∑i=1n(Yi−Ŷ i)

∂J∂w1=−1n∑i=1n(Yi−Ŷ i)Xi

These gradients are used to update the parameters until convergence is reached (i.e., when the changes in w0 and w1 become negligible).

Assumptions of Linear Regression

The following are some assumptions about the dataset that are made by the Linear Regression model −

Multi-collinearity − Linear regression model assumes that there is very little or no multi-collinearity in the data. Basically, multi-collinearity occurs when the independent variables or features have a dependency on them.

Auto-correlation − Another assumption the Linear regression model assumes is that there is very little or no auto-correlation in the data. Basically, auto-correlation occurs when there is dependency between residual errors.

Relationship between variables − Linear regression model assumes that the relationship between response and feature variables must be linear.

Violations of these assumptions can lead to biased or inefficient estimates. It is essential to validate these assumptions to ensure model accuracy.

Evaluation Metrics for Linear Regression

To assess the performance of a linear regression model, several evaluation metrics are used −

R-squared (R²) − It measures the proportion of the variance in the dependent variable that is predictable from the independent variables.

R2=1−∑(yi−ŷ i)2∑(yi−y¯)2

Mean Squared Error (MSE) − It measures an average of the sum of the squared difference between the predicted values and the actual values.

MSE=1n∑i=1n(yi−ŷ i)2

Root Mean Squared Error (RMSE) − It measures the square root of the MSE.

RMSE=MSE‾‾‾‾‾√

Mean Absolute Error (MAE) − It measures the average of the sum of the absolute values of the difference between the predicted values and the actual values.

MAE=1n∑i=1n|yi−ŷ i|

Applications of Linear Regression

1. Predictive Modeling

Linear regression is widely used for predictive modeling. For instance, in real estate, predicting house prices based on features such as size, location, and number of bedrooms can help buyers, sellers, and real estate agents make informed decisions.

2. Feature Selection

In multiple linear regression, analyzing the coefficients can help in feature selection. Features with small or zero coefficients might be considered less important and can be dropped to simplify the model.

3. Financial Forecasting

In finance, linear regression models predict stock prices, economic indicators, and market trends. Accurate forecasts can guide investment strategies and financial planning.

4. Risk Management

Linear regression helps in risk assessment by modeling the relationship between risk factors and financial metrics. For example, in insurance, it can model the relationship between policyholder characteristics and claim amounts.

Advantages of Linear Regression

• Interpretability − Linear regression is easy to understand, which is useful when explaining how a model makes decisions.
• Speed − Linear regression is faster to train than many other machine learning algorithms.
• Predictive analytics − Linear regression is a fundamental building block for predictive analytics.
• Linear relationships − Linear regression is a powerful statistical method for finding linear relationships between variables.
• Simplicity − Linear regression is simple to implement and interpret.
• Efficiency − Linear regression is efficient to compute.

Common Challenges with Linear Regression

1. Overfitting

Overfitting occurs when the regression model performs well on training data but lacks generalization on test data. Overfitting leads to poor prediction on new, unseen data.

2. Multicollinearity

When the dependent variables (predictor or feature variables) correlate, the situation is known as mutilcolinearty. In this, the estimates of the parameters (coefficients) can be unstable.

3. Outliers and Their Impact

Outliers can cause the regression line to be a poor fit for the majority of data points.

Polynomial Regression: An Alternate to Linear Regression

Polynomial Linear Regression is a type of regression analysis in which the relationship between the independent variable and the dependent variable is modeled as an n-th degree polynomial function. Polynomial regression allows for a more complex relationship between the variables to be captured beyond the linear relationship in Simple and Multiple Linear Regression.

What is Regression Analysis?

In machine learning, regression analysis is a statistical technique that predicts continuous numeric values based on the relationship between independent and dependent variables. The main goal of regression analysis is to plot a line or curve that best fit the data and to estimate how one variable affects another.

Regression analysis is a fundamental concept in machine learning and it is used in many applications such as forecasting, predictive analytics, etc.

In machine learning, regression is a type of supervised learning. The key objective of regression-based tasks is to predict output labels or responses, which are continuous numeric values, for the given input data. The output will be based on what the model has learned in the training phase.

Regression models use the input data features (independent variables) and their corresponding continuous numeric output values (dependent or outcome variables) to learn specific associations between inputs and corresponding outputs.

Terminologies Used In Regression Analysis

Let us understand some basic terminologies used in regression analysis before going into further detail. The following are some important terminologies −

• Independent Variables − These variables are used to predict the value of the dependent variable. These are also called predictors. In dataset, these are represented as features.
• Dependent Variables − These are the variables whose values we want to predict. These are the main factors in regression analysis. In dataset, these are represented as target variables
• Regression line − It is a straight line or curve that a regressor plots to fit the data points best.
• Overfitting and underfitting − Overfitting is when the regression model works well with the training dataset but not with the testing dataset. It’s also referred to as the problem of high variance. Underfitting is when the model doesn’t work well with training datasets. It’s also referred to as the problem of high bias.
• Outliers − These are data points that don’t fit the pattern of the rest of the data. They are the extremely high or extremely low values in the data set.
• Multicollinearity − multicollinearity occurs when independent variables (features) have dependency among them.

How Does Regression Work?

Regression in machine learning is a supervised learning. Basically, regression is a statistical technique that finds a relationship between dependent and independent variables. To implement regression in machine learning, a regression algorithm is trained with a labeled dataset. The dataset contains features (independent variables) and target values (dependent variable).

During the training phase, the regression algorithm learns the relation between independent variables (predictors) and dependent variables (target).

The regression models predict new values based on the learned relation between predictors and targets during the training.

Types of Regression in Machine Learning

Generally, the classification of regression methods is done based on the three metrics − the number of independent variables, type of dependent variables, and shape of the regression line.

There are numerous regression techniques used in machine learning. However, the following are commonly used types of regression −

• Linear Regression
• Logistic Regression
• Polynomial Regression
• Lasso Regression
• Ridge Regression
• Decision Tree Regression
• Random Forest Regression
• Support Vector Regression

Let’s discuss each type of regression in machine learning in detail.

1. Linear Regression

Linear regression is the most commonly used regression model in machine learning. It may be defined as the statistical model that analyzes the linear relationship between a dependent variable with a given set of independent variables. A linear relationship between variables means that when the value of one or more independent variables changes (increase or decrease), the value of the dependent variable will also change accordingly (increase or decrease).

Linear regression is further divided into two subcategories: simple linear regression and multiple linear regression (also known as multivariate linear regression).

In simple linear regression, a single independent variable (or predictor) is used to predict the dependent variable.

Mathematically, the simple linear regression can be represented as follows −

Y=mX+b

Where,

• Y is the dependent variable we are trying to predict.
• X is the dependent variable we are using to make predictions.
• m is the slope of the regression line, which represents the effect X has on Y.
• b is a constant known as the Y-intercept. If X=0, Y would be equal to b.

In multi-linear regression, multiple independent variables are used to predict the dependent variables.

We will learn linear regression in more detail in upcoming chapters.

2. Logistic Regression

Logistic regression is a popular machine learning algorithm used for predicting the probability of an event occurring.

Logistic regression is a generalized linear model where the target variable follows a Bernoulli distribution. Logistic regression uses a logistic function or logit function to learn a relationship between the independent variables (predictors) and dependent variables (target).

It maps the dependent variable as a sigmoid function of independent variables. The sigmoid function produces a probability between 0 and 1. The probability value is used to estimate the dependent variable’s value.

It is mostly used in binary classification problems, where the target variable is categorical with two classes. It models the probability of the target variable given the input features and predicts the class with the highest probability.

3. Polynomial Regression

Polynomial Linear Regression is a type of regression analysis in which the relationship between the independent variable and the dependent variable is modeled as an n-th degree polynomial function. Polynomial regression allows for a more complex relationship between the variables to be captured, beyond the linear relationship in Simple and Multiple Linear Regression.

Polynomial regression is one of the most widely used non-linear regressions. It is very useful because it can model non-linear relationships between predictors and targets, and also it is more sensitive to outliers.

4. Lasso Regression

Lasso regression is a regularization technique that uses a penalty to prevent overfitting and improve the accuracy of regression models. It performs L1 regularization. It modifies the loss function by adding the penalty (shrinkage quantity) equivalent to the summation of the absolute value of coefficients.

Lasso regression is often used to handle high dimensional and high correlation data.

5. Ridge Regression

Ridge regression is a statistical technique used in machine learning to prevent overfitting in linear regression models. It is used as a regularization technique that performs L2 regularization. It modifies the loss or cost function by adding the penalty (shrinkage quantity) equivalent to the square of the magnitude of coefficients.

Ridge regression helps to reduce model complexity and improve prediction accuracy. It is useful in developing many parameters with high weights. It is also well suited to datasets with more feature variables than a number of observations.

It also corrects the multicollinearity in regression analysis. Multicollinearity occurs when independent variables are dependent on each other.

6. Decision Tree Regression

Decision tree regression uses the decision tree algorithm to predict numerical values. The decision tree algorithm is a supervised machine learning algorithm that can be used for both classification and regression.

It is used to predict numerical values or continuous variables. It works by splitting the data into smaller subsets based on the values of the input features and assigning each subset a numerical value. So incrementally, it develops a decision tree

The tree fits local linear regressions that approximate a curve, and each leaf represents a numeric value. The algorithm tries to reduce the mean square error at each child node, which measures how much the predictions deviate from the original target.

The decision tree regression can be used in predicting stock prices or customer behavior etc.

7. Random Forest Regression

Random forest regression is a supervised machine learning algorithm that uses an ensemble of decision trees to predict continuous target variables. It uses a bagging technique that involves randomly selecting subsets of training data to build smaller decision trees. These smaller models are combined to form a random forest model that outputs a single prediction value.

The technique helps improve accuracy and reduce variance by combining the predictions from multiple decision trees.

8. Support Vector Regression

Support vector regression (SVR) is a machine learning algorithm that uses support vector machine to solve regression problems. It can learn non-linear relationships between the input data (feature variables) and output data (target values).

Support vector regression has many advantages. It can handle linear as well as non-linear relationships in datasets. It is resistant to outliers. It has high prediction accuracy.

Types of Regression Models

Regression models are of following two types −

Simple regression model − This is the most basic regression model in which predictions are formed from a single, univariate feature of the data.

Multiple regression model − As the name implies, in this regression model, the predictions are formed from multiple features of the data.

How to Select Best Regression Model?

You can consider factors like performance metrics, model complexity, interpretability, etc., to select the best regression model. Evaluate the model performance using metrics such as Mean Squared Error (MSE), Mean absolute error (MAE), R-squared, etc. Compare the performance of different models, such as linear regression, decision trees, random forests, etc., and choose a model that has the highest performance metrics, the lowest complexity, and the best interpretability.

Evaluation Metrics for Regression

Common evaluation/ performance metrics for regression models −

• Mean Absolute error (MAE) − It is the average of the absolute difference between predicted values and true values.
• Mean Squared error (MSE) − It is the average of the square of the difference between actual and estimated values.
• Median Absolute error − It is the median value of the absolute difference between predicted values and true values.
• Root mean square error (RMSE) − It is the square root value of the mean squared error (MSE).
• R² (coefficient of determination) Score − the best possible score is 1.0, and it can be negative (because the model can be arbitrarily worse).
• Mean absolute percentage error(MAPE) − It is the percentage equivalent of mean absolute error (MAE).

Applications of Regression in Machine Learning

The applications of ML regression algorithms are as follows −

Forecasting or Predictive analysis − One of the important uses of regression is forecasting or predictive analysis. For example, we can forecast GDP, oil prices, or, in simple words, the quantitative data that changes with the passage of time.

Optimization − We can optimize business processes with the help of regression. For example, a store manager can create a statistical model to understand the peak time of coming customers.

Error correction − In business, making correct decisions is equally important as optimizing the business process. Regression can help us to make correct decision as well as correct the already implemented decision.

Economics − It is the most used tool in economics. We can use regression to predict supply, demand, consumption, inventory investment, etc.

Finance − A financial company is always interested in minimizing the risk portfolio and wants to know the factors that affect the customers. All these can be predicted with the help of a regression model.

Building a Regressor in Python

Regressor model can be constructed from scratch in Python. Scikit-learn, a Python library for machine learning, can also be used to build a regressor in Python.

In the following example, we will be building a basic regression model that will fit a line to the data, i.e., linear regressor. The necessary steps for building a regressor in Python are as follows −

Step 1: Importing necessary python package

For building a regressor using scikit-learn, we need to import it along with other necessary packages. We can import the by using following script −

import numpy as np
from sklearn import linear_model
import sklearn.metrics as sm
import matplotlib.pyplot as plt

Step 2: Importing dataset

After importing necessary package, we need a dataset to build regression prediction model. We can import it from sklearn dataset or can use other one as per our requirement. We are going to use our saved input data. We can import it with the help of following script −

input=r'C:\linear.txt'

Next, we need to load this data. We are using np.loadtxt function to load it.

input_data = np.loadtxt(input, delimiter=',')
X, y = input_data[:,:-1], input_data[:,-1]

Step 3: Organizing data into training & testing sets

As we need to test our model on unseen data hence, we will divide our dataset into two parts: a training set and a test set. The following command will perform it −

training_samples =int(0.6*len(X))
testing_samples =len(X)- num_training
X_train, y_train = X[:training_samples], y[:training_samples]
X_test, y_test = X[training_samples:], y[training_samples:]

Step 4: Model evaluation & prediction

After dividing the data into training and testing we need to build the model. We will be using LineaRegression() function of Scikit-learn for this purpose. Following command will create a linear regressor object.

reg_linear = linear_model.LinearRegression()

Next, train this model with the training samples as follows −

reg_linear.fit(X_train, y_train)

Now, at last we need to do the prediction with the testing data.

y_test_pred = reg_linear.predict(X_test)

Step 5: Plot & visualization

After prediction, we can plot and visualize it with the help of following script −

plt.scatter(X_test, y_test, color ='red')
plt.plot(X_test, y_test_pred, color ='black', linewidth =2)
plt.xticks(())
plt.yticks(())
plt.show()

Output

In the above output, we can see the regression line between the data points.

Step 6: Performance computation

We can also compute the performance of our regression model with the help of various performance metrics as follows.

print("Regressor model performance:")print("Mean absolute error(MAE) =",round(sm.mean_absolute_error(y_test, y_test_pred),2))print("Mean squared error(MSE) =",round(sm.mean_squared_error(y_test, y_test_pred),2))print("Median absolute error =",round(sm.median_absolute_error(y_test, y_test_pred),2))print("Explain variance score =",round(sm.explained_variance_score(y_test, y_test_pred),2))print("R2 score =",round(sm.r2_score(y_test, y_test_pred),2))

Output

Regressor model performance:
Mean absolute error(MAE) = 1.78
Mean squared error(MSE) = 3.89
Median absolute error = 2.01
Explain variance score = -0.09
R2 score = -0.09