Deep Learning Jump Start

Deep Learning Jump Start
Michele Toni, 20 May 2017
Collegio Universitario Bertoni, Milano
Credits to Tommaso Matassini

About me
● M.Sc. in Computer Engineer for Business Administration at Università di Pisa
● 2 years as backend software developer.
● 1+ years as AI team member at Cynny.
More about Cynny: Morphcast site, Cynny Italian site, last public presentation

Today programme
● Deep Learning Showcase
● What is deep learning and how it works
○ Definitions
○ Neural Networks
○ Deep Learning
● How to start with deep learning
● Live demo: image classification Dogs Vs Cats with NVidia Digits

Deep Learning applications
Perception: recognizing what's in an image, what people are saying when they are
talking on their phone, helping robots explore the world and interact with it.
Images
Videos
Text
NLP
Music Other

Image Classification
[Try this online with CaffeJS]

Object Detection - 1/2
[Source: Awesome Deep Vision]

Object Detection - 2/2
[Source: DetectNet by Nvidia]

Image Segmentation
[Source: SegNet Online Demo]

Image Captioning
[Source: MS COCO Captioning Challenge 2015]

Image Captioning - Facebook easy approach
[Source: The Verge article]

Neural Art - DeepDream 1/3
[Source: Google Inceptionism]

Neural Art - DeepDream 2/3
[Source: L’Altra Toscana: Garfagnana]

Neural Art - Style Transfer 1/2
StyleContent

Neural Art - Style Transfer 2/2
[Try this: DeepDream, Prisma, Vinci]

Neural Art - Deep Photo Style Transfer
[Source Code and details: DeepPhoto Github]

Text Generation - Hemingway style
[Source: Hemingway style and Super Mario level generation]

Sequence2Sequence - Language translation
[Source: Attention and Memory in Deep Learning and NLP]

Sequence2Sequence - Chat bot
[Source: ChatBots with seq2seq, seq2seq Github]

Music Generation - Google Magenta and others
[Google Magenta Song, Fake Beatles Song, DeepBach by Sony CSL Music]

Generative Adversarial Network - pix2pix
[Try this: pix2pix Demo]

Generative Adversarial Network - CycleGAN
[Source code and details: CycleGAN Github]

Reinforcement learning - Atari Breakout
[Atari Breakout Video, Flappy Bird, OpenAI Gym]

Image analysis (old) approach: Computer Vision
HOG Face detector
[Source: dlib site]
Local Binary Pattern
[Source]

Why we use Deep Learning
[Source: Nervana Systems]

What is Deep Learning
And
how it works

AI Vs Machine Learning Vs Deep Learning
[Source: NVidia Blog]

Artificial Intelligence definition
Artificial intelligence (AI) is an area of computer science that emphasizes the
creation of intelligent machines that work and react like humans.
Some of the activities computers with artificial intelligence are designed for
include:
● Speech recognition
● Learning
● Planning
● Problem solving
[Source: Techopedia]

Machine Learning definition
Machine learning according to Arthur Samuel in 1959, gives "computers the
ability to learn without being explicitly programmed."
It explores the study and construction of algorithms that can learn from and
make predictions on data – such algorithms overcome following strictly
static program instructions by making data-driven predictions or decisions,
through building a model from sample inputs.
[Source: Wikipedia]

Types of learning
● Supervised learning: learn to predict an output when given an input
vector. We know the correct matching between input and output.
● Reinforcement learning: learn to select an action to maximize payoff.
● Unsupervised learning: discover a good internal representation of the
input. There is no known matching between input and output.
[Source: Geoffrey Hinton Neural Networks Coursera Course]

Types of learning: Supervised Learning
Each training case consists of an input vector x and a target ouput t.
● Regression: The target is a real number, e.g. the value of a stock, the
temperature.
● Classification: the target is a class label. E.g. from a given image tell if it
represents a cat or a dog.

Types of learning: Reinforcement Learning
● The output is an action or a sequence of actions and the only
supervisory signal is an occasional scalar reward. (No one tells which is
the correct action at each step, this has to be learned).
● The goal in selecting each action is to maximize the expected sum of
future rewards.
● Reinforcement learning is difficult, because the rewards can be delayed
and it is hard to know when we are wrong or right.

Types of learning: Unsupervised Learning 1/2
● Someone doesn’t consider unsupervised learning among the machine
learning techniques because it isn’t trained with input-output mapping.
● A typical example is clustering. E.g. to better visualize the inputs, to
study a problem, to prepare the data for a successive phase using
supervised or reinforcement learning.

Types of learning: Unsupervised Learning 2/2
[Source: T-Sne Visualization]

Biological neuron
[Source: Stanford CS231n Course]

Artificial neuron

Common non linear activation functions
[Source: Machine Learning for artists]

Examples of artificial neural network

What is a supervised learning model
Model: y = f(x; W)
f is a way to use numerical parameters W (called weights), to map each input x
into a predicted output y.
Learning: the procedure that adjusting the parameters W aims to reduce the
discrepancy between y (model output) and t (target output) for each training
sample.
Example of error function (MSE):

Training example with linear regression
Problem: we want to predict the price of a house knowing the area.
We have this data (our training dataset):
[Source: Visual and Interactive view of the basic of neural networks]

Linear regression model
[Source: Visual and Interactive view of the basic of neural networks, Wikipedia]
Error (MSE):

Linear regression model training - 1/2
Manual

Descending the gradient of the error (simplified)
[Source: Quora question]

Descending the gradient of the error (reality)
[Source]

Linear regression model training - 2/2
Gradient Descent in action

Deep Learning definition
“A family of learning methods that use deep architectures to learn high-level
feature representations”.
Examples of deep learning:
● Convolutional neural networks (for images)
● LSTM networks (for sequences like text and music)
[Source: Neural Machine Translation by Jointly Learning to Align and Translate]

Convolutional Neural Network layers
[Source: Stanford CS231n Course, in browser demo]

Convolution - 1/2
● Doesn’t matter where the cat is
● Different position, same cat
● We can share the weights!
[Source: Udacity Deep Learning Course by Google]

Convolution - 2/2
Graphical Demo

Pooling

CNN architecture: AlexNet (2012)
[Source: Visualize Neurons from Deep Models]

Convolutional Neural Networks recap
[Source: Siraj Raval Youtube Channel]

How to start with Deep Learning

What you need to train a CNN
DL Framework
Image Dataset Model structure

What you need to run a prediction
DL Framework
Model with trained weights

Datasets for specific tasks and challenges

Common models
View common model architectures online
Name Year Imagenet Top-5 Error #Parameters
Alexnet 2012 16,4 % 60 M
ZF 2013 11,7 % 16 M
GoogleNet 2014 6,7 % 7 M
VGG 2014 7,3 % 138 M
ResNet 2015 3,57 % 60 M
Inception-V4 2016 3,08 % 42 M
Squeezenet 2016 < 19,7 % ~ 1 M

Frameworks (very short list)
Name First release Main contributors Languages
Caffe / Caffe2 2013 / 2017 1 Berkeley University,
2 Facebook, Nvidia
Python, C++, Matlab
TensorFlow
(+ Keras API)
2015 Google Python, C++, Java, Go
Mxnet 2015 DMLC Python, Scala, Matlab,
C++, R, Julia, Go, JS
Torch 2002 (v1), 2015 (v7) Facebook, Twitter,
Google
LUA, C++

Live demo: Dogs Vs Cats
● Dataset: kaggle competition Dogs Vs Cats
● CNN Framework: Caffe + NVidia DIGITS
● Hardware: CUDA powered notebook (video card NVidia 970m)
● Model: Alexnet

Some additional AI playground links
DeepTraffic (simple self driving car training by MIT, with leaderboard)
AI Experiments with Google (e.g. AutoDraw, AI Duet)
TensorFlow Playground (try and visualize training)

Useful links to learn
Machine Learning is fun (intuitions behind how deep learning works)
Distill.pub (graphical and interactive paper publications)
Deep Learning Book by Yoshua Bengio
Neural Network Zoo (overview of neural networks architectures)
Dev Blog Nvidia - Parallel For All
9 Deep Learning Papers you need to know about
Awesome Deep Vision (Github papers repository)

Courses
Machine Learning by Andrew Ng (Coursera)
Neural Networks for Machine Learning (Coursera)
Deep Learning by Google (Udacity)
Deep Learning Nanodegree Foundation (Udacity)
Convolutional Neural Networks for Visual Recognition (Stanford)
Creative applications of deep learning with Tensorflow (Kadenze)

Other
Kaggle (machine learning competitions)
OpenAI Gym (environments and challenges to train reinforcement learning
models)
Google.ai

Deep Learning Jump Start

More Related Content

What's hot

Similar to Deep Learning Jump Start

Recently uploaded

Deep Learning Jump Start

Editor's Notes