Module5_LogisticRegression
References:
1. Ethem Alpaydin, "Introduction to
Machine Learning”, MIT Press, Prentice
Hall of India
Likelihood- vs. Discriminant-based
Classification
• Likelihood-based: Assume a model for p(x|Ci),
use Bayes’ rule to calculate P(Ci|x)
gi(x) = log P(Ci|x)
• Discriminant-based: Assume a model for gi(x|
Φi); no density estimation
– Estimating the boundaries is enough; need not be
accurately estimate the densities inside the
boundaries
Linear Discriminant
• Linear discriminant:
d
gi x | w i , w i 0 w Ti x w i 0 w ij x j w i 0
j 1
• Advantages:
– Simple: O(d) space/computation
– Knowledge extraction: Weighted sum of attributes;
positive/negative weights, magnitudes (credit scoring)
– Optimal when p(x|Ci) are Gaussian and when classes are
(almost) linearly separable
Generalized Linear Model
• Quadratic discriminant:
gi x | Wi , w i , w i 0 xT Wi x w Ti x w i 0
• Higher-order (product) terms:
z1 x1 , z 2 x 2 , z3 x12 , z 4 x 22 , z5 x1 x 2
Map from x to z using nonlinear basis functions and use a
linear discriminant in z-space
k
gi x w ij j x
j 1
Two Classes
gx g1 x g2 x
w1T x w10 w T2 x w 20
w1 w 2 T x w10 w 20
w T x w 0
C1 if gx 0
choose
C 2 otherwise
5
From Discriminants to Posteriors
y P C1 | x and P C 2 | x 1 y
y 0.5
choose C1 if y /1 y 1 and C 2 otherwise
log y / 1 y 0
Logistic Regression
• In statistics, the logistic model (or logit model) is a statistical model that models
the log-odds of an event as a linear combination of one or more independent
variables.
• Odds: Odds provide a measure of the probability of a particular outcome
• Odds Ratio: It is the ratio of something occurring to something not occurring. it
is different from probability as the probability is the ratio of something occurring
to everything that could possibly occur.
• Log-odds: The log-odds, also known as the logit function, is the natural logarithm
of the odds. In logistic regression, the log odds of the dependent variable are
modeled as a linear combination of the independent variables and the intercept.
• In regression analysis, logistic regression (or logit regression) estimates the
parameters of a logistic model (the coefficients in the linear or non linear
combinations).
Logistic Regression
• In binary logistic regression there is a single binary dependent variable,
coded by an indicator variable, where the two values are labeled "0" and
"1".
• The corresponding probability of the value labeled "1" can vary between
0 (certainly the value "0") and 1 (certainly the value "1").
• The function that converts log-odds to probability is the logistic function.
• The unit of measurement for the log-odds scale is called a logit (logistic
unit).
Logistic Regression
• Binary Logistic regression is used for binary classification where
we use sigmoid function, that takes input as independent
variables and produces a probability value between 0 and 1.
– For example, we have two classes Class 0 and Class 1 if the value of
the logistic function for an input is greater than 0.5 (threshold value)
then it belongs to Class 1 otherwise it belongs to Class 0. It’s referred
to as regression because it is the extension of linear regression but is
mainly used for classification problems.
• Logistic regression predicts the output of a categorical
dependent variable.
Logistic Function – Sigmoid
Function
• The sigmoid function is a mathematical function used to map
the predicted values to probabilities.
• It maps any real value into another value within a range of 0 and 1.
• The value of the logistic regression must be between 0 and 1, which
cannot go beyond this limit, so it forms a curve like the “S” form.
• The S-form curve is called the Sigmoid function or the logistic
function.
• In logistic regression, we use the concept of the threshold
value, which defines the probability of either 0 or 1.
• Such as values above the threshold value tends to 1, and a value
below the threshold values tends to 0.
Sigmoid (Logistic) Function
1. Calculate gx w T x w 0 and choose C1 if gx 0, or
2. Calculate y sigmoidw T x w 0 and choose C1 if y 0.5
Example
• Find the gender of the person with height=3
using Logistic Regression.
Height Gender
3 F
4 F
6 M
5 M
Example
• Find the gender of the person with height=3
using Logistic Regression.
– Let x be height and gender be y
– Let female be coded 1 and male be be 0
– Hence let us solve for Linear regression
parameters firstly using ordinary least squares
method
x y x2 x*y
3 1 9 3
4 1 16 4
6 0 36 0
5 0 25 0
Example
• Find the gender of the person with height=3 using Logistic
Regression.
• There slope,w1= -0.4 and intercept, w0 = 2.3
• Hence y=sigmoid (w1 x + w0)
• y= sigmoid(-0.4 * 3 + 2.3 ) = sigmoid(1.1)=0.75 (>0.5 , hence
output = 1 female)
x y x2 x*y
3 1 9 3
4 1 16 4
6 0 36 0
5 0 25 0
Other References
• https://www.simplilearn.com/tutorials/
machine-learning-tutorial/logistic-regression-
in-python
