Deep Learning

UNIT-I
 Neural Networks: History of Deep Learning,
 Deep Learning Success Stories
 McCulloch Pitts Neuron
 Thresholding Logic
 Perceptrons
 Perceptron Learning Algorithm
 Multilayer Perceptrons (MLPs)
 Representation Power of MLPs
 Sigmoid Neurons
 Gradient Descent
History Of Deep Learning

 Deep Learning is transforming the way machines understand, learn,

and interact with complex data.
 Deep learning mimics neural networks of the human brain.
 It enables computers to autonomously uncover patterns and make
informed decisions from vast amounts of unstructured data.
Historical trends in deep learning

https://medium.com/@lmpo/a-brief-history-of-ai-with-deep-learning-
26f7948bc87b
• Early Beginnings (1940s - 1960s) • 1943: The journey began
with Warren McCulloch and Walter Pitts' model of artificial
neurons, the McCulloch-Pitts neuron, which laid the foundation for
neural network theory. • 1957: Frank Rosenblatt introduced the
Perceptron, an early neural network model capable of learning
and recognizing patterns.

 The Winter of AI (1970s - 1980s) • Despite early

enthusiasm, neural networks faced challenges, including
computational limitations and the inability to train multi-layer
networks, leading to reduced interest in the field, known as the
"AI winter." • 1974: Paul Werbos developed backpropagation, a
key algorithm for training neural networks, but it remained
largely unnoticed until the mid-1980s.
 Revival and Growth (1980s - 1990s) • 1986: Geoffrey
Hinton, David Rumelhart, and Ronald Williams popularized
backpropagation, reviving interest in neural networks. • 1989:
Yann LeCun applied backpropagation to handwritten digit
recognition, leading to the development of Convolutional Neural
Networks (CNNs).
 The Emergence of Deep Learning (2000s) • 2006: Hinton
and his colleagues introduced the concept of deep belief
networks (DBNs), marking the formal beginning of deep
learning. • 2009: Fei-Fei Li's ImageNet project provided a large-
scale dataset for training deep learning models, fueling
advancements in computer vision.
 Breakthroughs and Dominance (2010s)
• 2012: Alex Krizhevsky, Ilya Sutskever, and Hinton won the
ImageNet competition with AlexNet, a deep CNN, demonstrating
the power of deep learning in image recognition.
• 2014: The introduction of Generative Adversarial Networks
(GANs) by Ian Goodfellow opened new possibilities in generative
modeling.
• 2015: Google's DeepMind developed AlphaGo, which defeated
the world champion Go player, showcasing deep learning's
potential in complex strategy games.
• 2016: The emergence of frameworks like TensorFlow and PyTorch
made deep learning more accessible to researchers and
practitioners.
 Recent Advances and Future Directions (2020s)
• 2020: OpenAI's GPT-3, a language model with 175
billion parameters, demonstrated the capabilities of deep
learning in natural language processing.
• Ongoing Research: Deep learning continues to evolve
with advancements in areas like reinforcement learning,
unsupervised learning, and multimodal learning.
Deep Learning Success Stories
1. Healthcare: Detecting Diseases with AI
Example: Google’s DeepMind and AI in Medical Imaging
 Faster and more accurate diagnoses
 Early disease detection saves lives
 Reduces workload for doctors

2. Self-Driving Cars: Tesla & Waymo

Example: Tesla Autopilot & Waymo’s AI Driver
 Safer driving with fewer accidents
 Reducing traffic congestion
 Potential for fully autonomous transportation
3. Natural Language Processing: ChatGPT
& Google Translate
Example: OpenAI’s ChatGPT, Google Translator
 Revolutionizing human-computer interaction
 Automating content creation and summarization
 Breaking language barriers worldwide
4. Gaming: AlphaGo & OpenAI Five
Example: AlphaGo defeating human Go champions
DeepMind’s AlphaGo beat world champion Go players
using deep reinforcement learning. Similarly, OpenAI
Five defeated professional Dota 2 players
 AI mastering complex strategy games
 Advancing reinforcement learning techniques
 Applications in real-world decision-making
5. Finance: Fraud Detection & Algorithmic
Trading
Example: AI in Fraud Detection
 Preventing financial fraud
 Enhancing security in banking
 Improving automated trading strategies
6. Entertainment: Deepfake Technology &
Personalized Recommendations
Example: Netflix & Spotify AI Recommendations
 Increased user engagement
 Better content discovery
 Improved customer retention for businesses

7. Climate Science: AI for Weather Forecasting

Example: NVIDIA’s FourCastNet
 Faster and more precise weather forecasting
 Better disaster preparedness
 Climate change research advancements
McCulloch pitts neuron model
 The McCulloch-Pitts Neuron is the first
computational model of a neuron.
 It can be divided into two parts:
 1. Aggregation: The neuron aggregates multiple
Boolean inputs (0 or 1).
 2. Threshold Decision: Based on the aggregated
value, the neuron makes a decision using a
threshold function.
The first computational model of a neuron was
proposed by Warren MuCulloch (neuroscientist)
and Walter Pitts (logician) in 1943.
Perceptron
Perceptron Learning Algorithm

The perceptron model consists of 4

steps:
 Input from other Neurons
 Weights and bias
 Net Sum
 Activation function

Step Activation Function:

Perceptron Learning Boolean AND Function
 Feedforward Neural Network

• Feedforward Neural Network (FNN) is a type

of artificial neural network in which
information flows in a single direction—from
the input layer through hidden layers to the
output layer—without loops or feedback.
• It is mainly used for pattern recognition
tasks like image and speech classification.
Structure of a Feedforward
Neural Network
Feedforward Neural Networks have a structured layered
design where data flows sequentially through each layer.
• Input Layer: The input layer consists of neurons that receive
the input data. Each neuron in the input layer represents a
feature of the input data.
• Hidden Layers: One or more hidden layers are placed
between the input and output layers. These layers are
responsible for learning the complex patterns in the data.
Each neuron in a hidden layer applies a weighted sum of
inputs followed by a non-linear activation function.
• Output Layer: The output layer provides the final output of
the network. The number of neurons in this layer
corresponds to the number of classes in a classification
problem or the number of outputs in a regression problem.
Feedforward Neural Network
Activation Functions

 Activation functions introduce non-linearity into the network

enabling it to learn and model complex data patterns.
 Common activation functions include:
Training a Feedforward Neural Network

 Training a Feedforward Neural Network involves adjusting the

weights of the neurons to minimize the error between the
predicted output and the actual output. This process is typically
performed using backpropagation and gradient descent.
 Forward Propagation: During forward propagation the input
data passes through the network and the output is calculated.
 Loss Calculation: The loss (or error) is calculated using a loss
function such as Mean Squared Error (MSE) for regression tasks
or Cross-Entropy Loss for classification tasks.
 Backpropagation: In backpropagation the error is propagated
back through the network to update the weights. The gradient of
the loss function with respect to each weight is calculated and
the weights are adjusted using gradient descent.
Backpropagation in Neural Network

 Back Propagation is also known as "Backward

Propagation of Errors" is a method used to train
neural network .
 Its goal is to reduce the difference between the
model’s predicted output and the actual output by
adjusting the weights and biases in the network.
 It works iteratively to adjust weights and bias to
minimize the cost function.
 It often uses optimization algorithms like gradient
descent or stochastic gradient descent.
Backpropagation in Neural
Network
Back Propagation plays a critical role in how neural
networks improve over time. Here's why:
 Efficient Weight Update: It computes the gradient
of the loss function with respect to each weight
using the chain rule making it possible to update
weights efficiently.
 Scalability: The Back Propagation algorithm scales
well to networks with multiple layers and complex
architectures making deep learning feasible.
 Automated Learning: With Back Propagation the
learning process becomes automated and the
model can adjust itself to optimize its performance.
Representation Power Of MLPS
 The representation power of a multilayer perceptron
(MLP) refers to its ability to approximate complex
functions and map inputs to outputs effectively.
 Here are key aspects of its representation power
1. Universal Approximation Theorem
 An MLP with at least one hidden layer and a nonlinear
activation function (such as ReLU, sigmoid, or tanh)
can approximate any continuous function to any desired
accuracy, given sufficient neurons in the hidden layer.
 This means that an MLP is a universal function
approximator, capable of representing highly complex
decision boundaries.
 This theorem provides a mathematical foundation
for why neural networks are capable of solving
complex problems across various domains like
image recognition, natural language
processing, and more.
2. Depth vs. Width
 Shallow MLPs (Single Hidden Layer): A single-
layer MLP with enough neurons can approximate
any function, but it may require an exponentially
large number of neurons.
 Deep MLPs (Multiple Hidden Layers): Adding
depth often allows a model to represent functions
more efficiently, requiring fewer neurons and
improving generalization.
3. Nonlinear Activations are Crucial
 Without nonlinear activation functions, an MLP is just
a linear transformation, no more powerful than a
single-layer perceptron.
 Activation functions like ReLU, tanh, or sigmoid
introduce nonlinearity, allowing the MLP to learn
complex representations.
4. Representation of Decision Boundaries
 A single-layer perceptron can only represent linearly
separable functions.
 An MLP with hidden layers can model nonlinear
decision boundaries, enabling it to solve problems like
XOR, image recognition, and natural language
processing
5. Expressive Power vs. Trainability
 While deep MLPs can represent complex functions,
training them effectively requires.
 Sufficient data to avoid over fitting.
 Proper weight initialization to avoid
vanishing/exploding gradients.
 Optimization techniques like batch normalization
and dropout to improve generalization.
Sigmoid Neurons

 Sigmoid neurons, also known as logistic neurons, are a type

of artificial neuron that uses the sigmoid function as its
activation function.
 This function squashes the output to a range between 0 and 1,
making it suitable for tasks like binary classification and as a
building block for deeper neural networks.
 It’s called a sigmoid neuron because its function’s output
forms an S-shaped curve when plotted on a graph; an S is
almost like a line that’s been squeezed into a smaller space.
Sigmoid Neurons
Gradient Descent

 Gradient Descent is an optimization algorithm used to minimize the loss function by

iteratively updating the weights in the direction of the negative gradient. Common
variants of gradient descent include:
 Batch Gradient Descent: Updates weights after computing the gradient over the
entire dataset.
 Stochastic Gradient Descent (SGD): Updates weights for each training example
individually.
 Mini-batch Gradient Descent: It Updates weights after computing the gradient over
a small batch of training examples.
Evaluation of Feedforward neural network
 Evaluating the performance of the trained model involves several metrics:
 Accuracy: The proportion of correctly classified instances out of the total instances.
 Precision: The ratio of true positive predictions to the total predicted positives.
 Recall: The ratio of true positive predictions to the actual positives.
 F1 Score: The harmonic mean of precision and recall, providing a balance between
the two.
 Confusion Matrix: A table used to describe the performance of a classification
model, showing the true positives, true negatives, false positives, and false negatives.