MACHINE LEARNING

TECHNIQUE: UNIT 2
RAJEEV KUMAR
A.P (SGU)
Research Domain: Li-Fi, IoT, ML, DL, Soft Computing, CV
UNIT-II

REGRESSION:
• Linear Regression
• Logistic Regression
BAYESIAN LEARNING:
• Bayes Theorem, Concept Learning
• Bayes Optimal Classifier
• Naïve Bayes Classifier, Bayesian Belief Networks, EM algorithm
SUPPORT VECTOR MACHINE:
• Introduction, Types of support vector kernel (Linear kernel, Polynomial kernel and Gaussian
kernel), Hyperplane - (Decision surface),
• Properties of SVM
• Issues in SVM
9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 2
Text books:
❖Tom M. Mitchell, ―Machine Learning, McGraw-Hill Education (India) Private Limited, 2013.
❖Ethem Alpaydin ―Introduction to Machine Learning (Adaptive Computation and Machine
Learning), MIT Press 2004.
❖Stephen Marsland ―Machine Learning: An Algorithmic Perspective, CRC Press, 2009
❖Bishop, C., Pattern Recognition and Machine Learning. Berlin: Springer-Verlag
❖M. Gopal, “Applied Machine Learning”, McGraw Hill Education

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 3

Regression

Regression analysis is a statistical method to model the

relationship between a dependent (target) and independent
(predictor) variables with one or more independent variables.

More specifically, Regression analysis helps us to understand

how the value of the dependent variable is changing
corresponding to an independent variable when other
independent variables are held fixed.

It predicts continuous/real values such as temperature, age,

salary, price, etc.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 4

Regression: Example

▪ Example: Suppose there is a marketing company A, who does various advertisement every
year and get sales on that. The below list shows the advertisement made by the company in the
last 5 years and the corresponding sales:
▪ Now, the company wants to do the advertisement of $200 in the year 2023 and wants to know
the prediction about the sales for this year. So to solve such type of prediction problems in
machine learning, we need regression analysis.

Regression is a Supervised Learning Technique which helps

in finding the correlation between variables and enables us to
predict the continuous output variable based on the one or more
predictor variables. It is mainly used for prediction,
forecasting, time series modeling, and determining the
causal-effect relationship between variables.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 5

Regression

▪ In Regression, we plot a graph between the variables which best fits the given datapoints, using
this plot, the machine learning model can make predictions about the data.
▪ In simple words, "Regression shows a line or curve that passes through all the datapoints
on target-predictor graph in such a way that the vertical distance between the datapoints
and the regression line is minimum."
▪ The distance between datapoints and line tells whether a model has captured a strong
relationship or not.
▪ Some examples of Regression can be as:
• Prediction of rain using temperature and other factors
• Determining Market trends
• Prediction of road accidents due to rash driving.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 6

Terminologies Related to the Regression Analysis

• Dependent Variable: The main factor in Regression analysis which we want to predict or
understand is called the dependent variable. It is also called target variable.
• Independent Variable: The factors which affect the dependent variables or which are used to
predict the values of the dependent variables are called independent variable, also called as
a predictor.
• Outliers: Outlier is an observation which contains either very low value or very high value in
comparison to other observed values. An outlier may hamper the result, so it should be avoided.
• Multicollinearity: If the independent variables are highly correlated with each other than other
variables, then such condition is called Multicollinearity. It should not be present in the dataset,
because it creates problem while ranking the most affecting variable.
• Underfitting and Overfitting: If our algorithm works well with the training dataset but not
well with test dataset, then such problem is called Overfitting. And if our algorithm does not
perform well even with training dataset, then such problem is called underfitting.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 7

Why do we use Regression Analysis?

▪ As mentioned above, Regression analysis helps in the prediction of a continuous variable.

There are various scenarios in the real world where we need some future predictions such as
weather condition, sales prediction, marketing trends, etc., for such case we need some
technology which can make predictions more accurately. So for such case we need Regression
analysis which is a statistical method and used in machine learning and data science. Below are
some other reasons for using Regression analysis:
• Regression estimates the relationship between the target and the independent variable.
• It is used to find the trends in data.
• It helps to predict real/continuous values.
• By performing the regression, we can confidently determine the most important factor, the
least important factor, and how each factor is affecting the other factors.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 8

Types of Regression

▪ There are various types of regressions which are used in data science and machine learning.
Each type has its own importance on different scenarios, but at the core, all the regression
methods analyze the effect of the independent variable on dependent variables.

Here we are discussing some important types of

regression which are given below:

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 9

Linear Regression

• Linear regression is a statistical regression method which is used for predictive analysis.
• It is one of the very simple and easy algorithms which works on regression and shows the
relationship between the continuous variables.
• It is used for solving the regression problem in machine learning.
• Linear regression shows the linear relationship between the independent variable (X-axis) and
the dependent variable (Y-axis), hence called linear regression.
• If there is only one input variable (x), then such linear regression is called simple linear
regression. And if there is more than one input variable, then such linear regression is
called multiple linear regression.
• The relationship between variables in the linear regression model can be explained using the
below image. Here we are predicting the salary of an employee on the basis of the year of
experience.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 11

Linear Regression

▪ Y= m X + C
Here, Y = dependent variables (target variables), X= Independent variables (predictor
variables), m and c are the linear coefficients
▪ Some popular applications of linear regression are:
• Analyzing trends and sales estimates
• Salary forecasting
• Real estate prediction
• Arriving at ETAs in traffic.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 12

Logistic Regression

• Logistic Regression is another Supervised Learning Algorithm which is used to solve the
classification problems. In classification problems, we have dependent variables in a binary or
discrete format such as 0 or 1.
• Logistic Regression Algorithm works with the categorical variable such as 0 or 1, Yes or No,
True or False, Spam or not spam, etc.
• It is a predictive analysis algorithm which works on the concept of probability.
• Logistic regression is a type of regression, but it is different from the linear regression
algorithm in the term how they are used.
• Logistic regression uses Sigmoid Function or logistic function which is a complex cost
function.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 13

Logistic Regression
• This Sigmoid Function is used to model the data in logistic regression. The function can be
represented as:
• f(x)= Output between the 0 and 1 value, x= input to the function, e= base of natural logarithm.
• When we provide the input values (data) to the function, it gives the S-curve as follows:
• It uses the concept of threshold levels, values above the threshold level are rounded up to 1, and
values below the threshold level are rounded up to 0.
▪ There are three types of logistic regression:
• Binary(0/1, pass/fail)
• Multi(cats, dogs, lions)
• Ordinal(low, medium, high)

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 14

Linear Regression

Linear Regression is one of the easiest and most popular Machine Learning algorithms.

It is a statistical method that is used for predictive analysis. Linear regression makes predictions for
continuous/real or numeric variables such as sales, salary, age, product price, etc.

Linear regression algorithm shows a linear relationship between a dependent (Y) and one or more
independent (X) variables, hence called as linear regression.

Since linear regression shows the linear relationship, which means it finds how the value of the
dependent variable is changing according to the value of the independent variable.

The linear regression model provides a sloped straight line representing the relationship between the
variables. Consider the below image:

9/29/2023 RAJEEV DHANDA, AP, CSE (SGU, GHAZIABAD) 16

Linear Regression

▪ Consider the below image:

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 17

Linear Regression

Mathematically, we can represent a Linear Regression as:

y= a0+a1x+ ε
Here,
Y= Dependent Variable (Target Variable)
X= Independent Variable (predictor Variable)
a0= intercept of the line (Gives an additional degree of freedom)
a1 = Linear regression coefficient (scale factor to each input value).
ε = random error
The values for x and y variables are training datasets for Linear Regression model representation.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 18

Types of Linear Regression

Linear regression can be further divided into two types of the algorithm:

Simple Linear Regression: If a single independent variable is used to predict

the value of a numerical dependent variable, then such a Linear Regression
algorithm is called Simple Linear Regression.

Multiple Linear regression: If more than one independent variable is used to

predict the value of a numerical dependent variable, then such a Linear
Regression algorithm is called Multiple Linear Regression.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 19

Linear Regression Line
▪ A linear line showing the relationship between the dependent and independent variables is
called a Regression Line. A regression line can show two types of relationship:

Positive Linear Relationship: If the dependent

variable increases on the Y-axis and independent
variable increases on X-axis, then such a relationship
is termed as a Positive linear relationship.

Negative Linear Relationship: If the dependent

variable decreases on the Y-axis and independent
variable increases on the X-axis, then such a
relationship is called a negative linear relationship.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 20

21 Linear Regression: Finding the Best Fit Line:

When working with linear regression, our main goal is to find the
best fit line that means the error between predicted values and
actual values should be minimized. The best fit line will have the
least error.

The different values for weights or the coefficient of lines (a0, a1)
gives a different line of regression, so we need to calculate the best
values for a0 and a1 to find the best fit line, so to calculate this we
use cost function.

Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 9/29/2023

9/29/2023 22

The different values for weights or coefficient of lines (a0,

a1) gives the different line of regression, and the cost
function is used to estimate the values of the coefficient
for the best fit line.

Cost function optimizes the regression coefficients or

Linear Regression: weights. It measures how a linear regression model is
Cost function performing.

We can use the cost function to find the accuracy of

the mapping function, which maps the input variable to
the output variable. This mapping function is also known
as Hypothesis function.

Rajeev Dhanda, AP, CSE (SGU, Ghaziabad)

Linear Regression

▪ For Linear Regression, we use the Mean Squared Error (MSE) cost function, which is the
average of squared error occurred between the predicted values and actual values. It can be
written as:
▪ For the above linear equation, MSE can be calculated as:

▪ Where,
▪ N=Total number of observation
▪ Yi = Actual value
▪ (a1xi+a0)= Predicted value.

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 23

LINEAR REGRESSION
Residuals: The distance between the actual value and predicted values is called residual. If
the observed points are far from the regression line, then the residual will be high, and so
cost function will high. If the scatter points are close to the regression line, then the residual
will be small and hence the cost function.

Gradient Descent:

• Gradient descent is used to minimize the MSE by calculating the gradient of the cost function.
• A regression model uses gradient descent to update the coefficients of the line by reducing the cost function.
• It is done by a random selection of values of coefficient and then iteratively update the values to reach the
minimum cost function.

9/29/2023 RAJEEV DHANDA, AP, CSE (SGU, GHAZIABAD) 24

Linear Regression: Model Performance

▪ The Goodness of fit determines how the line of regression fits the set of observations. The
process of finding the best model out of various models is called optimization. It can be
achieved by below method:
▪ R-squared method: R-squared is a statistical method that determines the goodness of fit.
• It measures the strength of the relationship between the dependent and independent
variables on a scale of 0-100%.
• The high value of R-square determines the less Difference between the “Predicted Values
and Actual Values” and hence represents a good model.
• It is also called a Coefficient of determination, or coefficient of multiple determination for
multiple regression.
• It can be calculated from the below formula:

9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 25

Assumption of Linear Regression
▪ Below are some important assumptions of Linear Regression. These are some formal checks while building a
Linear Regression model, which ensures to get the best possible result from the given dataset.
• Linear Relationship between the features and target: Linear regression assumes the linear relationship
between the dependent and independent variables.
• Small or no multicollinearity between the features: Multicollinearity means high-correlation between the
independent variables. Due to multicollinearity, it may difficult to find the true relationship between the
predictors and target variables. Or we can say, it is difficult to determine which predictor variable is affecting the
target variable and which is not. So, the model assumes either little or no multicollinearity between the features
or independent variables.
• Homoscedasticity Assumption: Homoscedasticity is a situation when the error term is the same for all the
values of independent variables. With homoscedasticity, there should be no clear pattern distribution of data in
the scatter plot.
• Normal distribution of Error Terms: Linear regression assumes that the error term should follow the normal
distribution pattern. If error terms are not normally distributed, then confidence intervals will become either too
wide or too narrow, which may cause difficulties in finding coefficients.
It can be checked using the q-q plot. If the plot shows a straight line without any deviation, which means the
error is normally distributed.
• No Autocorrelations: The linear regression model assumes no autocorrelation in error terms. If there will be
any correlation in the error term, then it will drastically reduce the accuracy of the model. Autocorrelation
usually occurs if there is a dependency between residual errors.
9/29/2023 Rajeev Dhanda, AP, CSE (SGU, Ghaziabad) 26