Pytorch
1
Agenda
• Introduction
• Basics of Pytorch
o Relevant Packages
o Computation graph
o Tensors and its operations
• Deep learning computations
o Defining a network
o Training a network
o Auto Differentiation
• Experiment walk through
3
Build a Neural Network from scratch in Python
4
Build a Neural Network from scratch in Python
What if you want to change your network by adding an additional layer?
What is the amount of code rework to be done approximately?
5
Build a Neural Network using Pytorch
What if you want to change your network by adding an additional layer?
What is the amount of code rework to done approximately?
6
Why Deep Learning Libraries?
• Easily build big computational graphs
• Easily compute gradients in computational graphs
• Run it all efficiently on GPU
Why Pytorch?
• PyTorch provides two high-level features
o Tensor computation with strong GPU acceleration
o Deep Neural Networks built on autodiff system
• Easy to implement, code, and debug
• More flexible due to its dynamic computational graph.
8
Neural networks library deeply integrated with autograd
PyTorch designed for maximumflexibility
• Important Packages:
a Tensor library like NumPy, with strong GPUsupport
torch. torch.
torch torch.nn
autograd optim
…
Auto differentiation library that Optimization package with standard optimization
supports all differentiable Tensor operations in methods such as SGD,RMSProp, LBFGS,Adametc.
torch
What is computational graph?
A computational graph is a way to represent a math function as a graph of
multiple tiny mathematical operations that leads to the same output
10
Basics of Pytorch
11
Tensor
• Tensors are similar to numpy’s ndarrays
• With the addition being that Tensors can also be used on a
GPU to accelerate computing
• Supports all the data types in numpy
• requires_grad
12
Basic operations
Initialize a tensor using a list
Initialize a 0-d tensor from scalar
Convert a 0-d tensor to python number
Always takes the datatype which as higher word size
Can specify the datatype while initializing
Just like numpy, creating a tensor by copying another
tensor results to pointing to same memory location
13
Basic operations
Generates a random tensor
Observe that for all the 3 operations
the outputs are same
14
Basic operations
Reshaping a tensor
Numpy -torch bridge
16
More Details
• https://pytorch.org/docs/stable/tensors.html
17
Define a Network
torch.nn
18
NN package
• Higher-level wrappers for working with neural nets
o Layers
o Activation Functions
o Loss functions
o Containers
• Gives flexibility to go beyond the prebuilt classes
19
MLP: Block Diagram
E.g. Cross Entropy Loss LOSS
Objective: Find out the best parameters
which will minimize the loss. Output Layer
Wn
Weight vector
Hidden Layers
W1
Input Layer
Label
Input 20
As Sequence of Layers
• Define our model as a
sequence of layers
21
As subclass of torch.nn.module
• Usually this approach is used to make user
defined/Complicated forward Propagations
possible
• Define a class with a method forward() (and a
constructor generally)
• __init__(): For defining / initializing the
necessary layers, parameters etc.
• forward(): Here we will define a flow of
input to output
22
Train a Network
23
Components Involved & Method
Network model
DataLoader
Loss Function
optimizer
24
Quick Glance into Ex4
• Expt-4
25
Auto Differentiation
26
Find the gradient of C w.r.t x,y,z
We can use Chain rule by Introducing intermediate
variables and by building a computational graph
28
Computational Graph: Numpy
Issues:
• No GPU Support
• Have to compute Gradients
on your Own
29
Computational Graph: Pytorch
Computation happens on
GPU, if available
Initialize a tensor with "requires_grad = True"
to Building a Computational Graph
Forward Pass
Backward to compute gradients
30
Numpy Pytorch Pytorch - GPU
31
PyTorch: Three Levels of Abstraction
• Tensor: Imperative ndarray, but runs on GPU
• Variable: Node in a computational graph; stores data and
gradient – “requires_grad = True”
• Module: A neural network layer; may store state or learnable
weights
32
Thanks!!
Questions?
34
Deep Learning Libraries
• Popular Libraries
PyTorch[Python] -- Facebook
Torch [Lua]
Theano [Python] -- University of Montreal
Caffe - UCBerkeley
TensorFlow – Google
Matconvnet – University of Oxford
Caffe 2
LeNet Torch Caffe PyTorch
1997 2009 2015 2017
1989 2002 2012 2016
LSTM Theano Tensor Caffe 2
Flow
