Background: Generative and
Logistic Discriminative Classifiers
Regressio
n
Logistic Regression
Important analytic tool in natural and
social sciences
Baseline supervised machine learning
tool for classification
Is also the foundation of neural
networks
Generative and Discriminative
Classifiers
Naive Bayes is a generative classifier
by contrast:
Logistic regression is a discriminative
classifier
Generative and Discriminative
Classifiers
Suppose we're distinguishing cat from dog images
imagenet imagenet
Generative Classifier:
• Build a model of what's in a cat image
• Knows about whiskers, ears, eyes
• Assigns a probability to any image:
• how cat-y is this image?
Also build a model for dog images
Now given a new image:
Run both models and see which one fits better
Discriminative Classifier
Just try to distinguish dogs from cats
Oh look, dogs have collars!
Let's ignore everything else
A generative models try to model how data is placed throughout the
space while discriminative models draw boundaries in the data
space.
A discriminative model directly learns the conditional probability
distribution P(y|x). Recall that generative model learns the joint
probability P(x,y) and then transform it to P(y|x) by using the Bayes
rule.
Naive bayes classifiers and hidden markov models are examples of
generative classifiers.
Logistic regression, SVM, and tree based classifiers (e.g. decision
tree) are examples of discriminative classifiers.
in
Generative vs Discriminative Classifiers
Naive Bayes
Logistic Regression
posterior
P(c|d)
8
Components of a probabilistic machine learning
classifier
Given m input/output pairs (x(i),y(i)):
The two phases of logistic regression
Training: we learn weights w and b using stochastic
gradient descent and cross-entropy loss.
Test: Given a test example x we compute p(y|x)
using learned weights w and b, and return
whichever label (y = 1 or y = 0) is higher probability
Background: Generative and
Logistic Discriminative Classifiers
Regressio
n
Classification in Logistic Regression
Logistic
Regressio
n
Classification Reminder
Positive/negative sentiment
Spam/not spam
Authorship attribution
(Hamilton or Madison?)
Alexander Hamilton
Text Classification: definition
Binary Classification in Logistic Regression
Features in logistic regression
• For feature xi, weight wi tells is how important is xi
• xi ="review contains ‘awesome’": wi = +10
• xj ="review contains ‘abysmal’": wj = -10
• xk =“review contains ‘mediocre’": wk = -2
Logistic Regression for one observation
x
How to do classification
For each feature xi, weight wi tells us importance of xi
◦ (Plus we'll have a bias b)
We'll sum up all the weighted features and the bias
If this sum is high, we say y=1; if low, then y=0
But we want a probabilistic classifier
We need to formalize “sum is high”.
We’d like a principled classifier that gives us a
probability, just like Naive Bayes did
We want a model that can tell us:
p(y=1|x; θ)
p(y=0|x; θ)
The problem: z isn't a probability, it's just a
number!
Solution: use a function of z that goes from 0 to 1
The very useful sigmoid or logistic function
21
Idea of logistic regression
We’ll compute w∙x+b
And then we’ll pass it through the
sigmoid function:
σ(w∙x+b)
And we'll just treat it as a probability
Making probabilities with sigmoids
By the way:
Because
Turning a probability into a classifier
0.5 here is called the decision boundary
The probabilistic classifier
P(y=1)
wx + b
Turning a probability into a classifier
if w∙x+b > 0
if w∙x+b ≤ 0
Classification in Logistic Regression
Logistic
Regressio
n
Logistic Regression: a text example
Logistic on sentiment classification
Regressio
n
Sentiment example: does y=1 or y=0?
It's hokey . There are virtually no surprises , and the writing is second-rate .
So why was it so enjoyable ? For one thing , the cast is
great . Another nice touch is the music . I was overcome with the urge to get off
the couch and start dancing . It sucked me in , and it'll do the same to you .
30
31
for any classification task: period
disambiguation
End of sentence
This ends in a period.
The house at 465 Main St. is new.
Not end
32
Classifying sentiment for input x
Suppose w =
b = 0.1 33
Classifying sentiment for input x
34
Classification in (binary) logistic regression:
summary
Given:
◦ a set of classes: (+ sentiment,- sentiment)
◦ a vector x of features [x1, x2, …, xn]
◦ x1= count( "awesome")
◦ x2 = log(number of words in review)
◦ A vector w of weights [w1, w2, …, wn]
◦ wi for each feature fi
Logistic Regression: a text example
Logistic on sentiment classification
Regressio
n
Learning: Cross-Entropy Loss
Logistic
Regressio
n
Wait, where did the W’s come from?
38
Learning components
A loss function:
◦ cross-entropy loss
An optimization algorithm:
◦ stochastic gradient descent
Intuition of negative log likelihood loss
= cross-entropy loss
A case of conditional maximum likelihood
estimation
We choose the parameters w,b that maximize
• the log probability
• of the true y labels in the training data
• given the observations x
Deriving cross-entropy loss for a single
observation x
Deriving cross-entropy loss for a single observation
x
Goal: maximize probability of the correct label p(y|x)
Maximize:
Now take the log of both sides (mathematically handy)
Maximize:
Whatever values maximize log p(y|x) will also maximize
p(y|x)
Deriving cross-entropy loss for a single observation
x
Goal: maximize probability of the correct label p(y|x)
Maximize:
Minimize:
Let's see if this works for our sentiment
example
We want loss to be:
• smaller if the model estimate is close to correct
• bigger if model is confused
Let's first suppose the true label of this is y=1 (positive)
It's hokey . There are virtually no surprises , and the writing is second-rate
. So why was it so enjoyable ? For one thing , the cast is great .
Another nice touch is the music . I was overcome with the urge to get off
the couch and start dancing . It sucked me in , and it'll do the same to you
.
Let's see if this works for our sentiment
example
True value is y=1. How well is our model doing?
Pretty well! What's the loss?
Let's see if this works for our sentiment
example
Suppose true value instead was y=0.
What's the loss?
Let's see if this works for our sentiment
example
The loss when model was right (if true y=1)
Is lower than the loss when model was wrong (if true y=0):
Sure enough, loss was bigger when model was wrong!
Cross-Entropy Loss
Logistic
Regressio
n
Stochastic Gradient Descent
Logistic
Regressio
n
Our goal: minimize the loss
Intuition of gradient descent
How do I get to the bottom of this river canyon?
Look around me 360∘
Find the direction of
steepest slope down
x Go that way
Our goal: minimize the loss
For logistic regression, loss function is convex
• A convex function has just one minimum
• Gradient descent starting from any point is
guaranteed to find the minimum
• (Loss for neural networks is non-convex)
Let's first visualize for a single scalar w
Q: Given current w, should we make it bigger or smaller?
A: Move w in the reverse direction from the slope of the function
Let's first visualize for a single scalar w
Q: Given current w, should we make it bigger or smaller?
A: Move w in the reverse direction from the slope of the function
So we'll move positive
Let's first visualize for a single scalar w
Q: Given current w, should we make it bigger or smaller?
A: Move w in the reverse direction from the slope of the function
So we'll move positive
Gradients
The gradient of a function of many variables is a
vector pointing in the direction of the greatest
increase in a function.
Gradient Descent: Find the gradient of the loss
function at the current point and move in the
opposite direction.
How much do we move in that direction ?
Now let's consider N dimensions
We want to know where in the N-dimensional
space (of the N parameters that make up θ ) we
should move.
The gradient is just such a vector; it expresses the
directional components of the sharpest slope along
each of the N dimensions.
Imagine 2 dimensions, w and b
Visualizing the
gradient vector at
the red point
It has two
dimensions shown
in the x-y plane
Real gradients
Are much longer; lots and lots of weights
For each dimension wi the gradient component i
tells us the slope with respect to that variable.
◦ “How much would a small change in wi influence the
total loss function L?”
◦ We express the slope as a partial derivative ∂ of the loss
∂wi
The gradient is then defined as a vector of these
partials.
What are these partial derivatives for logistic
regression?
The loss function
The elegant derivative of this function (see textbook 5.8 for derivation)
Hyperparameters
The learning rate η is a hyperparameter
◦ too high: the learner will take big steps and overshoot
◦ too low: the learner will take too long
Hyperparameters:
• Briefly, a special kind of parameter for an ML model
• Instead of being learned by algorithm from
supervision (like regular parameters), they are
chosen by algorithm designer.
Stochastic Gradient Descent
Logistic
Regressio
n
Stochastic Gradient Descent:
Logistic An example and more details
Regressio
n
Working through an example
One step of gradient descent
A mini-sentiment example, where the true y=1 (positive)
Two features:
x1 = 3 (count of positive lexicon words)
x2 = 2 (count of negative lexicon words)
Assume 3 parameters (2 weights and 1 bias) in Θ0 are zero:
w1 = w2 = b = 0
η = 0.1
Example of gradient descent w1 = w2 = b = 0;
Update step for update θ is: x1 = 3; x2 = 2
where
Gradient vector has 3 dimensions:
Example of gradient descent w1 = w2 = b = 0;
Update step for update θ is: x1 = 3; x2 = 2
where
Gradient vector has 3 dimensions:
Example of gradient descent w1 = w2 = b = 0;
Update step for update θ is: x1 = 3; x2 = 2
where
Gradient vector has 3 dimensions:
Example of gradient descent w1 = w2 = b = 0;
Update step for update θ is: x1 = 3; x2 = 2
where
Gradient vector has 3 dimensions:
Example of gradient descent
Now that we have a gradient, we compute the new parameter vector
θ1 by moving θ0 in the opposite direction from the gradient:
η = 0.1;
Example of gradient descent
Now that we have a gradient, we compute the new parameter vector
θ1 by moving θ0 in the opposite direction from the gradient:
η = 0.1;
Example of gradient descent
Now that we have a gradient, we compute the new parameter vector
θ1 by moving θ0 in the opposite direction from the gradient:
η = 0.1;
Example of gradient descent
Now that we have a gradient, we compute the new parameter vector
θ1 by moving θ0 in the opposite direction from the gradient:
η = 0.1;
Note that enough negative examples would eventually make w2 negative
Mini-batch training
Stochastic gradient descent chooses a single
random example at a time.
That can result in choppy movements
More common to compute gradient over batches of
training instances.
Batch training: entire dataset
Mini-batch training: m examples (512, or 1024)
Stochastic Gradient Descent:
Logistic An example and more details
Regressio
n
