1
Neural networks for
classification and
regression
Outline
▪ Why deep learning?
▪ Neural networks
• activation functions
▪ Training
• Back propogation
• Stochastic gradient descent
▪ Quiz Review and answer to some questions from exercises/problem sets
3
Introduction Linear regression Logistic regression
Feature engineering Data statistics Naive Bayes
KNN Clustering Dimensionality reduction
Neural networks Convolutional neural Decision-trees
networks
Background for Neural networks
Review of supervised learning
i d
feature vectors, independent variables x E IR i = 1
,
,
2
...,
N
m
i
Labels, dependent variables, target, outcome Y
i
e IR F
J
+ 11 ,
2
, . . .,
k)
Training data/set/example S & xi ,
yig for supervised leary
,
\xih . for unsupervised leaving
Sample, sample point ,
data point
↑
xi
ji)
&
S ,
X
Why deep learning?
Logistic regression review
Logistic regression for d-dimensional data:
Input: Activation
Input Weight Output
function
Weights:
Bias: x1
w1
1
Logistic: σ(z) = x2
1 + e −z w2
… wT x + b σ ŷ
wd
xd
Why deep learning?
Limitations of logistic regression
Issue: Logistic regression performs badly on non-
linearly separable data
Potential fix: Use feature engineering to make data
linearly separable, then use logistic regression
However:
▪ Features that linearly separate the data can be
hard to find manually, specially in high dimension
Why deep learning?
From logistic regression to neural networks
Neural networks have been successful in learning complex, non-linear functions
Why deep learning?
New way to approach ML
Before deep learning:
Data Features Model
Hand-design the features
Deep learning:
Data Features Model
Deep neural networks derive
useful features from the data!
Neural networks
Neural networks
Representation
weights weights
“2-layer Neural Net”, or “3-layer Neural Net”, or
“1-hidden-layer Neural Net” “2-hidden-layer Neural Net”
“Fully-connected” layers
Each neuron of a layer is connected
to all neurons of the following layer
Neural networks
Inside a neuron
g = Activation function
Connections to biological neurons
Source: towardsdatascience.com
Applications - nowadays everywhere!
Applications - examples in IGM
Neural networks
Representation
x1
[l]
ai Layer
Node in layer
x = x2 Shape (3, 1)
x3
Neural networks
Representation [1]
w1
x1
[l]
ai Layer
Node in layer
x = x2
x3
Weight vector for first node of first layer:
[1]
w1,1
[1] [1]
w1 = w1,2
1]
w1,3
Neural networks
Representation [1]
w1
x1
[l]
ai Layer
Node in layer
x = x2
x3
Weight vector for first node of first layer:
[1]
w1,1
[1] [1]
w1 = w1,2
1]
w1,3
[1] [1]T [1]
z1 = w1 x + b1
[1] [1]T [1]
z2 = w2 x + b2
[1] [1]T [1]
z3 = w3 x + b3
[1] [1]T [1]
z4 = w4 x + b4
Neural networks
Representation [1]
w1
x1
[l]
ai Layer
Node in layer
x = x2
x3
Weight vector for first node of first layer:
[1]
w1,1
[1] [1]
w1 = w1,2 Shape (3, 1)
1]
w1,3
[1] [1]T [1] [1] [1] [1]
z1 = w1 x + b1 a1 = g (z1 ) nonlinear transformaten
[1] [1]T [1] [1] [1] [1] of
z2 = w2 x + b2 a2 = g (z2 ) x
Apply activation
[1] [1]T [1] [1] [1] [1]
z3 = w3 x + b3 a3 = g (z3 )
[1]
z4 = [1]T [1]
w4 x + b4
g] [1]
a4 = [1] [1]
g (z4 )
Neural networks
Representation [1]
w1
x1
[l]
ai Layer
Node in layer
x = x2
x3
Vector notation:
[1]
b1
⋮ ⋮ ⋮ ⋮ [1]
[1] [1] [1] [1] [1] b2
W = w1 w2 w3 w4 b [1]
= [1]
b3
⋮ ⋮ ⋮ ⋮
[1]
b4
Shape (3, 4)
[1] [1]T [1] [1] [1] [1]
z =W x+b Apply activation a = g (z )
Activation functions
NN - Activation Function
Introduction
[2] [2]T [1] [1]T [1] [2]
ŷ = g (W g (W x+b )+b )
Q: What happens if we remove the activations?
NN - Activation Function
Introduction
[2] [2]T [1] [1]T [1] [2]
ŷ = g (W g (W x+b )+b )
Q: What happens if we remove the activations?
̂y = W[2]T (W[1]T x + b[1]) + b[2]
[2]T [1]T [2]T [1] [2]
ŷ = W W x+W b +b
T [2]T [1]T [2]T [1] [2]
Define W′ =W W Define b′ = W b +b
T
ŷ = W′ x + b′
A: We end up with a linear classifier!
NN - Activation functions
Introduction
To model a nonlinear problem:
▪ Pass the output of each neuron through a nonlinear function,
called activation function
▪ Connection to neuron firing in brain
Some well-known activation functions:
▪ Sigmoid
▪ Tanh
▪ ReLU
NN - Activation functions
Overview
Sigmoid (σ): Tanh:
▪ Squashes input in a [0, 1] range ▪ Squashes input in a [-1, 1] range
▪ Approximately nullifies gradient (for “large” positive or ▪ Like sigmoid, nullifies gradient (for “large” positive or
negative inputs) -> vanishing gradient problem negative inputs)
▪ rarely used except for final layer of binary classification ▪ Zero-centered, preferable over sigmoid as an activation
network ▪ Rarely used in practice (ReLU is more popular)
NN - Activation functions
Overview
Rectified Linear Unit (ReLU): Leaky ReLU:
▪ Easily computed, simple gradient ▪ Attempts to fix “dying ReLU” problem by having a
▪ Greatly accelerates convergence of gradient descent small negative slope for x < 0.
▪ Saturates in only one direction, suffers less from ▪ Leaky ReLU and other ReLU variants (ELU,
vanishing gradient problem SELU, GELU, Swish, etc…) are sometimes used
over ReLU
▪ commonly used in practice
NN - Activation functions
Derivatives
Sigmoid: Rectified Linear Unit (ReLU):
{0 if x ≤ 0
1 x if x > 0
σ(x) = ReLU(x) = = max(0, x)
1 + e −x
d
{0 if x < 0
σ(x) = σ(x)(1 − σ(x)) d 1 if x > 0
dx ReLU(x) =
dx
Tanh:
x −x Note: Derivative of ReLU is undefined for
e −e
tanh(x) = x x = 0. By convention, it is set to 0.
e + e −x
d 2
tanh(x) = 1 − tanh (x)
dx
Training neural nets
Neural networks
Training
Forward pass of 2 layer NN (for a single example):
[1] [1]T [1]
z =W x+b
[1] [1] [1]
a = g (z )
[2] [2]T [1] [2]
z =W a +b
[2] [2] [2]
ŷ = a = g (z )
[2] [2]T [1] [1]T [1] [2]
ŷ = g (W g (W x+b )+b )
[1] [2]
W W
Neural networks
Training
Forward pass of 2 layer NN (for a single sample):
[2] [2]T [1] [1]T [1] [2]
ŷ = g (W g (W x+b )+b )
To train, we need a loss function: L(y,̂ y)
Using that loss function, we want to update
[1] [1] [2] [2]
W ,b ,W ,b
using gradient descent.
[1] [2]
W W
Loss function
Loss function
Gradient descent
∂L ∂L
Need to compute: ,
∂W[i] ∂b[i]
=> Gradient of loss with respect
to weights
Once gradients are computed,
update weights with:
[i] ∂L
[i]
▪ W := W − α
∂W[i]
[i] [i] ∂L
▪ b := b − α [i]
∂b
where α is the learning rate
Neural networks
Forward / Backward pass
Forward pass: Compute the output of a neural network for a given input
Backward pass: Compute derivatives of the network parameters given the output
During training, you need both the forward pass and the backward pass.
During inference, you only need the forward pass.
Inference: the process of using a trained machine learning model for prediction
Computing gradients
Back propagation
Computing gradients
Back propagation
Neural networks
Forward pass
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
[1] [1] [1]
a = g (z )
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
[1] [1] [1]
a = g (z )
[2] [2]T [1] [2]
z =W a +b
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
[1] [1] [1]
a = g (z )
[2] [2]T [1] [2]
z =W a +b
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
[1] [1] [1]
a = g (z )
[2] [2]T [1] [2]
z =W a +b
[2] [2] [2]
a = g (z )
Neural networks
Forward pass
[1] [1]T [1]
z =W x+b
[1] [1] [1]
a = g (z )
[2] [2]T [1] [2]
z =W a +b
[2] [2] [2]
a = g (z )
[2]
ŷ = a
Neural networks
Forward pass
Forward pass of this 2-layer NN:
[1] [1]T [1]
z =W x+b
[1] [1] [1]
a = g (z )
[2] [2]T [1] [2]
z =W a +b
[2] [2] [2]
a = g (z )
[2]
ŷ = a
Rewriting it in one equation:
[2] [2]T [1] [1]T [1] [2]
ŷ = g (W g (W x+b )+b )
Stochastic gradient descent
Mini-batch stochastic gradient descent
Problems with training
Recap on training a neural network
Loop:
1. Sample a batch of data
2. Forward pass to get the loss
3. Backward pass to calculate gradient
4. Update parameters using the gradient
▪ Forward pass computes result of an operation and save any intermediates
needed for gradient computation in memory
▪ Backward pass applies the chain rule to compute the gradient of the loss
function with respect to the inputs
Deep learning frameworks
Overview
Deep learning frameworks are used to efficiently define and train neural networks
• Support for many types of layers, activations, loss functions, optimizers, …
• Backpropagation computed automatically (e.g. loss.backward() in PyTorch)
• GPU support for faster training
Most popular frameworks today:
• PyTorch (https://pytorch.org)
• TensorFlow (https://www.tensorflow.org/)
Deep learning frameworks
Implementing a simple neural network in PyTorch
Extensions of feedforward neural networks
▪ Convolutional neural networks (next lecture)
▪ Recurrent neural networks (relevance for control system)
Python exercises
▪ You will create a neural network for hand-written digit classification
▪ Training data is based on MNIST dataset: online dataset of 70,000 images containing hand-written digits
▪ Training neural networks is time (and energy) consuming
28 x 28 =
784
▪ We will use Google Colab as it has access to faster processing
• GPU (Graphic processing units)
• TPU (Tensor processing units)
↑
D
Brief review of last lecture
PCA
k-means
Questions from problem set
Data covariance matrix
