Vivekanand Education Society’s

Institute of Technology,
Chembur, Mumbai

ARTIFICIAL NEURAL NETWORK AND FUZZY LOGIC

FA C U LT Y N A M E : A S H W I N I S AWA N T
A S S I S TA N T P R O F E S S O R
D E PA RT M E N T O F E L E C T R O N I C S & T E L E C O M M U N I C AT I O N E N G I N E E R I N G
 Artificial Neural Networks

Ashwini Sawant
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 2
Agenda
 History of Artificial Neural Networks
 What is an Artificial Neural Networks?
 How it works?
 Learning
 Learning paradigms
 Supervised learning
 Unsupervised learning
 Reinforcement learning
 Applications areas
 Advantages and Disadvantages
 History of the Artificial Neural Networks
 History of the ANNs stems from the 1940s, the decade of the first electronic
computer.
 However, the first important step took place in 1957 when Rosenblatt introduced
the first concrete neural model, the perceptron. Rosenblatt also took part in
constructing the first successful neurocomputer, the Mark I Perceptron. After this,
the development of ANNs has proceeded as described in Figure.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4

History of the Artificial Neural Networks
 Rosenblatt's original perceptron model contained only one layer. From this, a
multi-layered model was derived in 1960. At first, the use of the multi-layer
perceptron (MLP) was complicated by the lack of a appropriate learning
algorithm.
 In 1974, Werbos came to introduce a so-called backpropagation algorithm for the
three-layered perceptron network.
 History of the Artificial Neural Networks
 In 1986, The application area of the MLP networks remained rather limited until the
breakthrough when a general back propagation algorithm for a multi-layered
perceptron was introduced by Rummelhart and Mclelland.
 in 1982, Hopfield brought out his idea of a neural network. Unlike the neurons in
MLP, the Hopfield network consists of only one layer whose neurons are fully
connected with each other.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6

History of the Artificial Neural Networks
 Since then, new versions of the Hopfield network have been developed. The
Boltzmann machine has been influenced by both the Hopfield network and
the MLP.
 History of the Artificial Neural Networks
 in 1988, Radial Basis Function (RBF) networks were first introduced by
Broomhead & Lowe. Although the basic idea of RBF was developed 30
years ago under the name method of potential function, the work by
Broomhead & Lowe opened a new frontier in the neural network
community.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 8

History of the Artificial Neural Networks
 in 1982, A totally unique kind of network model is the Self-Organizing Map
(SOM) introduced by Kohonen. SOM is a certain kind of topological map
which organizes itself based on the input patterns that it is trained with. The
SOM originated from the LVQ (Learning Vector Quantization) network the
underlying idea of which was also Kohonen's in 1972.
 History of Artificial Neural Networks
Since then, research on artificial neural networks has
remained active, leading to many new network types, as
well as hybrid algorithms and hardware for neural
information processing.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 10

 Evolution of neural networks
Year Neural network Designer Description
1943 McCulloch and Pitts McCulloch and Pitts Arrangement of
neuron neurons is combination
of logic gate. Unique
feature is thresh hold
1949 Hebb network Hebb If two neurons are
active, then their
connection strengths
should be increased.
1958,1959,1962,1988,1 Perceptron Frank Rosenblatt, Block, Weights are adjusted to
960 Adaline Minsky and Papert reduce the difference
Widrow and Hoff between the net input
to the output unit and
the desired output

11
 Contd…
1972 Kohonen self- Kohonen Inputs are clustered
organizing feature to obtain a fired
map output neuron.
1982, Hopfield network John Hopfield and Based on fixed
1984, Tank weights.
1985, Can act as associative
1986, memory nets
1987
1986 Back propagation Rumelhart, Hinton i) Multilayered
network and Williams ii) Error propagated
backward from
output to the
hidden units

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 1212

 Contd..
1988 Counter Grossberg Similar to kohonen
propagation network
network
1987-1990 Adaptive resonance Carpenter and Designed for binary
Theory(ART) Grossberg and analog inputs.
1988 Radial basis function Broomhead and Resemble back
network Lowe propagation
network , but
activation function
used is Gaussian
function
1988 Neo cognitron Fukushima For character
recognition.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 1313
 Artificial Neural Network
 An artificial neural network consists of a pool of simple
processing units which communicate by sending signals to
each other over a large number of weighted connections.
 A Neural Network is a machine that is designed to model
the way in which the brain performs a particular task or
function of interest; the network is usually implemented
by using electronic components or is simulated in software
on a digital computer.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 14

 Artificial Neural Network
 A set of major aspects of a parallel distributed model include:
▪ a set of processing units (cells).
▪ a state of activation for every unit, which equivalent to the output of the unit.
▪ connections between the units. Generally each connection is defined by a weight.
▪ a propagation rule, which determines the effective input of a unit from its external inputs.
▪ an activation function, which determines the new level of activation based on the
effective input and the current activation.
▪ an external input for each unit.
▪ a method for information gathering (the learning rule).
▪ an environment within which the system must operate, providing input signals and _ if
necessary _ error signals.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 15

 Computers vs. Neural Networks
“Standard” Computers Neural Networks

 one CPU highly parallel processing

fast processing units slow processing units

reliable units unreliable units

static infrastructure dynamic infrastructure

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 16

 Brain Vs computer
Term Brain Computer
Speed Execution time is few milliseconds Execution time is few nano seconds

Processing Perform massive parallel operations Perform several parallel operations

simultaneously simultaneously. It is faster the
biological neuron
Size and complexity Number of Neuron is 1011 and It depends on the chosen application
number of interconnections is 1015. and network designer.
So complexity of brain is higher than
computer

Storage capacity i) Information is stored in i) Stored in continuous memory

interconnections or in synapse location.
strength. ii) Overloading may destroy older
ii) New information is stored without locations.
destroying old one. iii) Can be easily retrieved
iii) Sometimes fails to recollect
information

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 1717

Contd…
Tolerance i) Fault tolerant i) No fault tolerance
ii) Store and retrieve ii) Information corrupted
information even if the network
interconnections fails connections
iii) Accept redundancies disconnected.
iii) No redundancies
Control mechanism Depends on active CPU
chemicals and neuron Control mechanism is very
connections are strong or simple
weak

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 1818

 Why Artificial Neural Networks?
There are two basic reasons why we are interested in building
artificial neural networks (ANNs):

• Technical viewpoint: Some problems such as

character recognition or the prediction of future
states of a system require massively parallel and
adaptive processing.

• Biological viewpoint: ANNs can be used to

replicate and simulate components of the human
(or animal) brain, thereby giving us insight into
natural information processing.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 19
 Artificial Neural Networks
• The “building blocks” of neural networks are the neurons.
• In technical systems, we also refer to them as units or nodes.
• Basically, each neuron
 receives input from many other neurons.
 changes its internal state (activation) based on the current input.
 sends one output signal to many other neurons, possibly including
its input neurons (recurrent network).

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 20

 Artificial Neural Networks
• Information is transmitted as a series of electric impulses, so-
called spikes.

• The frequency and phase of these spikes encodes the

information.

• In biological systems, one neuron can be connected to as many

as 10,000 other neurons.

• Usually, a neuron receives its information from other neurons

in a confined area, its so-called receptive field.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 21

 How do ANNs work?
 An neural network (ANN) is either a hardware
artificial
implementation or a computer program which strives to simulate the
information processing capabilities of its biological exemplar. ANNs are
typically composed of a great number of interconnected artificial neurons.
The artificial neurons are simplified models of their biological
counterparts.
 ANN is a technique for solving problems by constructing software that
works like our brains.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 22

 How do our brains work?
▪ The Brain is A massively parallel information processing
system.
▪ Our brains are a huge network of processing elements. A
typical brain contains a network of 10 billion neurons.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 23

 How do our brains work?
▪ A processing element

Dendrites: Input
Cell body: Processor
Synaptic: Link
Axon: Output
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 24
 How do our brains work?
▪ A processing element

A neuron is connected to other neurons through about 10,000

synapses

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 25

 How do our brains work?
▪ A processing element

A neuron receives input from other neurons. Inputs are combined.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 26

 How do our brains work?
▪ A processing element

Once input exceeds a critical level, the neuron discharges a spike ‐

an electrical pulse that travels from the body, down the axon, to
the next neuron(s)
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 27
 How do our brains work?
▪ A processing element

The axon endings almost touch the dendrites or cell body of the
next neuron.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 28

 How do our brains work?
▪ A processing element

Transmission of an electrical signal from one neuron to the next is

effected by neurotransmitters.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 29

 How do our brains work?
▪ A processing element

Neurotransmitters are chemicals which are released from the first neuron
and which bind to the
Second.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 30
 How do our brains work?
▪ A processing element

This link is called a synapse. The strength of the signal that

reaches the next neuron depends on factors such as the amount of
neurotransmitter available.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 31
 How do ANNs work?

An artificial neuron is an imitation of a human neuron

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 32
 How do ANNs work?
• Now, let us have a look at the model of an artificial neuron.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 33

 How do ANNs work?
............
Input xm x2 x1

Processing ∑
∑= X1+X2 + ….+Xm =y

Output y

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 34

 How do ANNs work?
Not all inputs are equal
............
xm x2 x1
Input
wm ..... w2 w1
weights
Processing ∑ ∑= X1w1+X2w2 + ….+Xmwm
=y

Output y

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 35

 How do ANNs work?
The signal is not passed down to the
next neuron verbatim
............
xm x2 x1
Input
wm ..... w2 w1
weights
Processing ∑
Transfer Function
f(vk)
(Activation Function)

Output y
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 36
 The output is a function of the input, that is affected by the weights,
and the transfer functions

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 37

 Artificial Neural Networks
 An ANN can:
1. compute any computable function, by the appropriate
selection of the network topology and weights values.
2. learn from experience!
▪ Specifically, by trial‐and‐error

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 38

 Learning by trial‐and‐error
Continuous process of:
➢Trial:
Processing an input to produce an output (In terms of ANN: Compute
the output function of a given input)
➢Evaluate:
Evaluating this output by comparing the actual output with
the expected output.
➢Adjust:
Adjust the weights.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 39

 Neural Networks
-N E U R A L NE T W O R K WA S I N S P I R E D B Y T H E D E S I G N A N D F U N C T I O N I N G O F
HUMAN BRAIN AND COMPONENTS.
-D E F I N I T I O N :
-I N F O R M AT I O N P R O C E S S I N G M O D E L T H AT I S I N S P I R E D B Y T H E WAY
B I O L O G I C A L N E RV O U S S Y S T E M ( I . E ) T H E B R A I N , P R O C E S S I N F O R M AT I O N .
-A N N I S C OM P O S E D O F L A R G E N U M B E R O F H I G H LY I N T E R C O NN E C T E D
P R O C E S S I N G E L E M E N T S ( N E U R O N S ) W O RK I N G I N U N I S O N TO S O LV E
PROBLEMS.
-I T I S C ON F I G U R E D F O R S P E C I A L A P P L I C AT I O N S U C H A S PAT T E R N
R E C O G N I T I O N A N D D ATA C L A S S I F I C AT I O N T H R O U G H A L E A R N I N G
PROCESS.
-8 5 - 9 0 % A C C U R AT E .

40
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 40
 Advantages of Neural Networks
• A Neural Network can be an “expert” in analyzing the category of information given to it.
• Answers “ what-if” questions
• Adaptive learning
– Ability to learn how to do tasks based on the data given for training or initial experience.
• Self organization
– Creates its own organization or representation of information it receives during learning time.
• Real time operation
– Computations can be carried out in parallel.
• Fault tolerance via redundant information coding
– Partial destruction of neural network cause degradation of performance.
– In some cases, it can be retained even after major network damage.
• In future, it can also used to give spoken words as instructions for machine.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4141

 This figure shows the multi
disciplinary point of view of Neural
Networks

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4242

 Application Scope of Neural
Networks
1. Animal behavior 12. Air traffic control
2. Appraisal and valuation of property, etc., 13. Fraud detection
3. Betting on horse races, stock markets 14. Hand writing and typewriting
4. Criminal sentencing 15. Lake water levels
5. Complex physical and chemical process 16. Machinery controls
6. Data mining, cleaning and validation 17. Medical diagnosis
7. Direct mail advertisers 18. Music composition
8. Echo patterns 19. Photos and finger prints
9. Economic modeling 20. Recipes and chemical formulation
10. Employee hiring 21. Traffic flows
11. Expert consultants 22. Weather prediction

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 43

 Fuzzy Logic
Lofti Zadeh, Professor at University of California.
An organized method for dealing with imprecise data
Fuzzy logic includes 0 and 1 as extreme cases of truth (or "the state of matters" or "fact") but
also includes the various states of truth in between so that, for example, the result of a
comparison between two things could be not "tall" or "short" but ".38 of tallness.“
Allows partial membership
Implemented in small, embedded micro controllers to large , networked, multichannel PC or
work station.
Can be implemented in hardware, software or in both.
It mimics how a person would make decisions.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4444

 Genetic algorithm
How genes of parents combine to form those of their children.
Create an initial population of individuals representing possible solutions to
solve a problem
Individual characters determine whether they are less or more fit to the
population
The more fit members will take high probability.
It is very effective in finding optimal or near optimal solutions.
Generate and test strategy.
Differ from normal optimization and search procedures in:
◦ Work with coding of the parameter set
◦ Work with multiple points
◦ Search via sampling( a blind search)
◦ Search using stochastic opeartors
In business, scientific and engineering circles, etc.,

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4545

 Hybrid System
Three types:-
Neuro Fuzzy hybrid system
◦ Combination of fuzzy set theory and neural networks
◦ Fuzzy system deal with explicit knowledge that can be explained and understood
◦ Neural network deal with implicit knowledge acquired by learning
◦ Advantages are:
◦ Handle any kind of information
◦ Manage imprecise, partial, vague or imperfect information
◦ Resolve conflicts by collaboration and aggregation.
◦ Self learning, self organizing and self tuning capability
◦ No need for prior knowledge of relationship of data
◦ Mimic human decision making system
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4646
Contd..
Neuron genetic hybrid system
◦ Topology optimization
◦ Select a topology for ANN , common one is back propagation
◦ Genetic algorithm training
◦ Learning of ANN is formulated ad weight optimization problem, usually mean
squared error as a fitness measure
◦ Control parameter optimization
◦ Learning rate, momentum rate, tolerance level. Etc., are optimized using GA.
Fuzzy genetic hybrid system
◦ Creating the classification rules for a fuzzy system where objects are classified
by linguistic terms.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4747
 Soft computing
Two major problem solving techniques are:
◦ Hard computing
◦ Deals with precise model where
accurate solutions are achieved.
◦ Soft computing
◦ deals with approximate model to give
solution for complex problems
◦ Prof. Lotfi Zadeh introduced it.
◦ Ultimate goal-emulate the human mind
◦ It is a combination of GA, Neural
Network and FL.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4848

 Artificial Neural Network : An
Introduction
 Resembles the characteristic of biological neural network.
 Nodes – interconnected processing elements (units or neurons)
 Neuron is connected to other by a connection link.
 Each connection link is associated with weight which has information about
the input signal.
 ANN processing elements are called as neurons or artificial neurons , since
they have the capability to model networks of original neurons as found in
brain.
 Internal state of neuron is called activation or activity level of neuron, which
is the function of the inputs the neurons receives.
 Neuron can send only one signal at a time.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4949
 Basic Operation of a Neural Net
X1 and X2 – input neurons.
Y- output neuron
Weighted interconnection links- W1
and W2.
Net input calculation is :

Output is :

Output= function

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5050

Contd…
The function to be applied over the net input is called activation function.
Weight involved in ANN is equal to the slope of linear straight line (y=mx).

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5151

 Biological Neural Network
Has three main parts
◦ Soma or cell body-where cell nucleus is located
◦ Dendrites-where the nerve is connected to the cell body
◦ Axon-which carries the impulses of the neuron
Electric impulse is passed between synapse and dendrites.
Synapse- Axon split into strands and strands terminates into small bulb like organs called
as synapse.
It is a chemical process which results in increase /decrease in the electric potential inside
the body of the receiving cell.
If the electric potential reaches a thresh hold value, receiving cell fires & pulse / action
potential of fixed strength and duration is send through the axon to synaptic junction of the
cell.
After that, cell has to wait for a period called refractory period.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5252
Contd..
In this model net input is calculated by

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5353

Terminology Relation Between Biological And
Artificial Neuron
Biological Neuron Artificial Neuron
Cell Neuron
Dendrites Weights or
interconnections
Soma Net input
Axon Output

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5454

 Characteristics of ANN:
Neurally implemented mathematical model
Large number of processing elements called neurons exists here.
Interconnections with weighted linkage hold informative knowledge.
Input signals arrive at processing elements through connections and
connecting weights.
Processing elements can learn, recall and generalize from the given data.
Computational power is determined by the collective behavior of neurons.
◦ ANN is a connection models, parallel distributed processing models, self-
organizing systems, neuro-computing systems and neuro morphic system.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5555
 Basic models of ann
Models are based on three entities
◦ The model’s synaptic interconnections.
◦ The training or learning rules adopted for updating and adjusting the connection weights.
◦ Their activation functions
The arrangement of neurons to form layers and the connection pattern formed within and
between layers is called the network architecture.
Five types:
◦ Single layer feed forward network
◦ Multilayer feed-forward network
◦ Single node with its own feedback
◦ Single-layer recurrent network
◦ Multilayer recurrent network

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5656

 Single layer Feed- Forward Network
Layer is formed by taking processing
elements and combining it with other
processing elements.
Input and output are linked with each
other
Inputs are connected to the
processing nodes with various
weights, resulting in series of outputs
one per node.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5757

 Multilayer feed-forward network
Formed by the interconnection of several
layers.
Input layer receives input and buffers input
signal.
Output layer generated output.
Layer between input and output is called
hidden layer.
Hidden layer is internal to the network.
Zero to several hidden layers in a network.
More the hidden layer, more is the complexity
of network, but efficient output is produced.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5858
 Feed back network
If no neuron in the output layer is an input to a node in the
same layer / proceeding layer – feed forward network.
If outputs are directed back as input to the processing
elements in the same layer/proceeding layer –feedback
network.
If the output are directed back to the input of the same layer
then it is lateral feedback.
Recurrent networks are networks with feedback networks
with closed loop.
Fig 2.8 (A) –simple recurrent neural network having a
single neuron with feedback to itself.
Fig 2.9 – single layer network with feedback from output
can be directed to processing element itself or to other
processing element/both.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5959
Maxnet –competitive interconnections having fixed
weights.
On-center-off-surround/lateral inhibiton structure –
each processing neuron receives two different classes of
inputs- “excitatory” input from nearby processing
elements & “ inhibitory” elements from more distantly
located precessing elements. This type of
interconnection is shown below

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6060

 Multilayer Recurrent Network
Processing element output can be directed
back to the nodes in the preceding layer,
forming a multilayer recurrent network.
Processing element output can be directed
to processing element itself or to other
processing element in the same layer.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6161

learning
Two broad kinds of learning in ANNs is :
i) parameter learning – updates connecting weights in a neural net.
ii) Structure learning – focus on change in the network.
Apart from these, learning in ANN is classified into three categories as
◦ i) supervised learning
◦ ii) unsupervised learning
◦ iii) reinforcement learning

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6262

 Supervised learning
Learning with the help of a teacher.
Example : learning process of a small child.
 Child doesn’t know read/write.
 Their each & every action is supervised by a teacher
In ANN, each input vector requires a corresponding target vector, which
represents the desired output.
The input vector along with target vector is called training pair.
The input vector results in output vector.
The actual output vector is compared with desired output vector.
If there is a difference means an error signal is generated by the
network.
It is used for adjustment of weights until actual output matches desired
output.
20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6363
 Unsupervised learning
Learning is performed without the help of a teacher.
Example: tadpole – learn to swim by itself.
In ANN, during training process, network receives input
patterns and organize it to form clusters.
From the Fig. it is observed that no feedback is applied
from environment to inform what output should be or
whether they are correct.
The network itself discover patterns, regularities, features/
categories from the input data and relations for the input
data over the output.
Exact clusters are formed by discovering similarities &
dissimilarities so called as self – organizing.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 64

64
 Reinforcement learning
Similar to supervised learning.
Learning based on critic information is called
reinforcement learning & the feedback sent is
called reinforcement signal.
The network receives some feedback from the
environment.
Feedback is only evaluative.
The external reinforcement signals are processed
in the critic signal generator, and the obtained
critic signals are sent to the ANN for adjustment
of weights properly to get critic feedback in
future.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6565

 Activation functions
To make work more efficient and for exact output, some force or activation is given.
Like that, activation function is applied over the net input to calculate the output of an ANN.
Information processing of processing element has two major parts: input and output.
An integration function (f) is associated with input of processing element.
Several activation functions are there.
1. Identity function:
 it is a linear function which is defined as
f(x) =x for all x
 The output is same as the input.
2. Binary step function
 it is defined as

where θ represents thresh hold value.

It is used in single layer nets to convert the net input to an output that is bianary. ( 0 or 1)

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6666

 Contd..
3. Bipolar step function:
• It is defined as

• where θ represents threshold value.

• used in single layer nets to convert the net input to an output that is bipolar (+1 or -1).
4. Sigmoid function
used in Back propagation nets.
Two types:
a) binary sigmoid function
-logistic sigmoid function or unipolar sigmoid function.
-it is defined as
where λ – steepness parameter.
-The derivative of this function is
f’(x) = λ f(x)[1-f(x)]. The range of sigmoid function is 0 to 1.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6767

Contd..
b) Bipolar sigmoid function

where λ- steepness parameter and the sigmoid range is between -1 and +1.
- The derivative of this function can be
[1+f(x)][1-f(x)]
- It is closely related to hyberbolic tangent function, which is written as

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6868

Contd..
The derivative of the hyberbolic tangent function is
h’(x)= [1+h(x))][1-h(x)]
5. Ramp function

The graphical representation of all these function is given in the upcoming Figure

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6969

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7070
 Important terminologies
Weight
 The weight contain information about the input signal.
 It is used by the net to solve the problem.
 It is represented in terms of matrix & called as connection matrix.
 If weight matrix W contains all the elements of an ANN, then the set of all W matrices will determine the
set of all possible information processing configuration.
 The ANN can be realized by finding an appropriate matrix W.
 Weight encode long-term memory (LTM) and the activation states of network encode short-term memory
(STM) in a neural network.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 71

71
Contd..
Bias
◦ Bias has an impact in calculating net input.
◦ Bias is included by adding x0 to the input vector x.
◦ The net output is calculated by

◦ The bias is of two types

◦ Positive bias
◦ Increase the net input
◦ Negative bias
◦ Decrease the net input

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7272

Contd..
Threshold
◦ It is a set value based upon which the final output is calculated.
◦ Calculated net input and threshold is compared to get the network output.
◦ The activation function of threshold is defined as

◦ where θ is the fixed threshold value

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7373

Contd..
Learning rate
 Denoted by α.
 Control the amount of weight adjustment at each step of training.
 The learning rate range from 0 to 1.
 Determine the rate of learning at each step
Momentum Factor
 Convergence is made faster if a momentum factor is added to the weight updation process.
 Done in back propagation network.
Vigilance parameter
 Denoted by ρ.
 Used in Adaptive Resonance Theory (ART) network.
 Used to control the degree of similarity.
 Ranges from 0.7 to 1 to perform useful work in controlling the number of clusters.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7474

 Mcculloch-pitts neuron
Discovered in 1943.
Usually called as M-P neuron.
M-P neurons are connected by directed weighted paths.
Activation of M-P neurons is binary (i.e) at any time step the neuron may fire or may not fire.
Weights associated with communication links may be excitatory(wgts are
positive)/inhibitory(wgts are negative).
Threshold plays major role here. There is a fixed threshold for each neuron and if the net input to
the neuron is greater than the threshold then the neuron fires.
They are widely used in logic functions.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7575

 Contd…
A simple M-P neuron is shown in the
figure.
It is excitatory with weight (w>0) /
inhibitory with weight –p (p<0).
In the Fig., inputs from x1 to xn
possess excitatory weighted
connection and Xn+1 to xn+m has
inhibitory weighted interconnections.
Since the firing of neuron is based on
threshold, activation function is
defined as

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 76

76
Contd…
For inhibition to be absolute, the threshold with the activation function should satisfy the
following condition:
θ >nw –p

Output will fire if it receives “k” or more excitatory inputs but no inhibitory inputs where
kw≥θ>(k-1) w
- The M-P neuron has no particular training algorithm.
- An analysis is performed to determine the weights and the threshold.
- It is used as a building block where any function or phenomenon is modeled based on a
logic function.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7777

Linear separability
It is a concept wherein the separation of the input space into regions is based on whether the
network response is positive or negative.
A decision line is drawn to separate positive or negative response.
The decision line is also called as decision-making line or decision-support line or linear-
separable line.
The net input calculation to the output unit is given as

The region which is called as decision boundary is determined by the relation

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7878

 Contd..
Consider a network having positive
response in the first quadrant and
negative response in all other
quadrants with either binary or bipolar
data.
Decision line is drawn separating two
regions as shown in Fig.
Using bipolar data representation,
missing data can be distinguished
from mistaken data. Hence bipolar
data is better than binary data.
Missing values are represented by 0
and mistakes by reversing the input
values from +1 to -1 or vice versa.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7979

 Hebb network
Donald Hebb stated in 1949 that “ In brain, the learning is performed by the change in the synaptic gap”.
When an axon of cell A is near enough to excite cell B, and repeatedly or permanently takes place in firing it, some
growth process or metabolic change takes place in one or both the cells such than A’s efficiency, as one of the cells
firing B, is increased.
According to Hebb rule, the weight vector is found to increase proportionately to the product of the input and the
learning signal.
In Hebb learning, two interconnected neurons are ‘on’ simultaneously.
The weight update in Hebb rule is given by
 Wi(new) = wi (old)+ xi y.

It is suited more for bipolar data.

If binary data is used, the weight updation formula cannot distinguish two conditions namely
 A training pair in which an input unit is “on” and the target value is “off”
 A training pair in which both the input unit and the target value is “off”.

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 8080

 Flowchart of training
algorithm
Steps:
◦ 0: First initialize the weights.
◦ 1: Steps 2-4 have to be performed for each input training vector and target output pair, s:t
◦ 2: Input activations are set. The activation function for input layer is identity function.
◦ Xi =Si for i=1 to n
◦ 3: Output activations are set.
◦ 4: Weight adjustment and bias adjustments are performed.
◦ Wi(new) = wi(old)+xiy
◦ b(new)=b(old)+y
◦ In step 4, the weight updation formula can be written in vector form as
◦ w(new)=w(old)+y.
◦ Change in weight is expressed as
◦ Δw=xy
Hence,
w(new)=w(old)+Δw
Hebb rule is used for pattern association, pattern categorization, pattern classification
and over a range of other areas

20/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 8181