SOFT COMPUTING-SWE1011

MODULE-2: Memory Models

 SYLLABUS

• Pattern association
• Auto associative memory models
• Hetero associative memory models
• BAM
• Hopfield network
PATTERN ASSOCIATION
 PATTERN ASSOCIATION

• Pattern association is carried out efficiently by associative

memory networks
• An associative network is a single-layer net in which the
weights are determined in such a way that the net can store a
set of pattern associations.
• Each association is an input-output vector ( pair s:t)
• The two main algorithms used for training a pattern
association network are
• The Hebb rule
• The outer products rule
 ASSOCIATIVE MEMORY NETWORK

• It stores a set of patterns as memories

• When it is presented with a key pattern, it responds by
producing one of the stored patterns
• The selected pattern resembles or relates to the key pattern
• The recall is through association of the key pattern
• These types of memories are called content- addressable
memories (CAM)
• In digital computers we have address- addressable memories
• As in RAM/ROM it is also a matrix memory
 ASSOCIATIVE MEMORY NETWORK CONTD…

• The input data is correlated with that of the stored data in

CAM
• The stored patterns must be unique (That is different
patterns are stored in different locations)
• Each data stored has an address
• If multiple copies are stored the data will be correct. But the
address will be ambiguous
• The concept behind this search is to output any one or all
stored items which matches the given search argument
• The stored data is retrieved completely or partially
 ASSOCIATIVE MEMORY NETWORK CONTD…

• Two types of such networks are there

• AUTO ASSOCIATIVE
• HETERO ASSOCIATIVE
• If the output vector is same as the input vectors it is Auto
Associative
• If the output vectors are different from the input vectors it is
Hetero Associative
• To find the similarity we use the Hamming distance
 HAMMING DISTANCE

• Suppose we have two vectors,

x = ( x1 , x2 ,...xn )T and y = (y1 , y2 ,...yn )T
• Then the Hamming Distance is defined as
n
 xi − yi , if xi , yi  {0,1};
 i =1
HD( x, y ) =  n
1
 2 
xi − yi , if xi , yi  {−1,1}.
i =1

• The hamming distance basically denotes the number of

places where the two vectors differ
 TRAINING ALGORITHMS FOR PATTERN
ASSOCIATION
• There are two algorithms for training of pattern association
nets
1. Hebb Rule
2. Outer Products Rule
• Hebb Rule
• This is widely used for finding the weights of an associative
memory neural net
• Here, the weights are updated until there is no weight
change
 ALGORITHMIC STEPS FOR HEBB RULE

• STEP 0: Set all the initial weights to zero

( wij = 0, i = 1, 2...n; j = 1, 2,...m )
• STEP 1: For each training target input output vector pairs (s:t)
perform steps 2 to 4
• STEP 2: Activate the input layer units to current training input
( xi = si , i = 1, 2,...n)
• STEP 3: Activate the output layer units to the target output
( y j = t j , j = 1, 2,...m)
• STEP 4: Start the weight adjustments
wij (new) = wij (old) + x i  y j , i = 1, 2,...n; j = 1, 2,...m
 OUTER PRODUCT RULE

• Here, we have
• Input: s = ( s1 , s2 ,..., si ,...sn )
• Output: t = (t1 , t2 ,...t j ,...tm )
• The outer product is defined as: The product of the two
matrices: S = sT and T = t
• So, we get the weight matrix W as:
 s1   s1t1 . s1t j s1tm 
.  
W=S.T =   =  
 si  . t1 . t j . tm   si t1 si t j si tm 
   
.  
 sn 
 snt1 sn t j sntm 

 OUTER PRODUCT RULE CONTD…

• In case of a set of patterns, s(p): t(p), p=1,…P; we have

s( p) = ( s1 ( p),..., si ( p),..., sn ( p))
t( p) = (t1 ( p),..., t i ( p),..., t m ( p))

• For the weight matrix W = ( wij )

P
wij =  siT ( p).t j ( p), i = 1, 2,...n; j = 1, 2,...m.
p =1
 AUTOASSOCIATIVE MEMORY NETWORK

• The training input and target output vectors are the same
• The determination of weights of the association net is called
storing of vectors
• The vectors that have been stored can be retrieved from
distorted (noisy) input if the input is sufficiently similar to it
• The net’s performance is based on its ability to reproduce a
stored pattern from a noisy input

• NOTE: The weights in the diagonal can be set to ‘0’. These

nets are called auto-associative nets with no self-connection
 ARCHITECTURE OF AUTOASSOCIATIVE
MEMORY NETWORK
• A
x1 w11 y1
X1 Y1
wi1
wn1

w1i
xi yi
Xi wii Yi
wni

w1n
win

xn yn
Xn wnn Yn
 TRAINING ALGORITHM

• This is same as that for the Hebb rule. Except that there are
same number of output units as the number of input units
• STEP 0: Initialize all the weights to ‘0’
• ( wij = 0, i = 1, 2,...n; j = 1, 2,...n )
• STEP 1: For each of the vector that has to be stored, perform
• steps 2 to 4
• STEP 2: Activate each of the input unit ( xi = si , i = 1, 2,...n )
• STEP 3: Activate each of the output units (y j = s j , j = 1,2,...n )
• STEP 4: Adjust the weights, i, j = 1,2,…n;
wij (new) = wij (old ) + xi . y j = wij (old ) + si .s j
 TESTING ALGORITHM

• The associative memory neural network can be used to

determine whether the given input vector is a “known” one
or an unknown one
• The net is said to recognize a vector if the net produces a
pattern of activation as output, which is the same as the one
given as input
• STEP 0: Set the weights obtained from any of the two
methods described above
• STEP 1: For each of the testing input vector perform steps 2-4
• STEP 2: Set the activations of the input unit as equal to that of
the input vector
 TESTING ALGORITHM CONTD…

• STEP 3: Calculate the net input to each of the output unit:

n
( yin ) j =  xi .wij , j = 1, 2,...n
i =1

• STEP 4: Calculate the output by applying the activation over

the net input:
1, if ( yin ) j  0;
y j = f (( yin ) j ) = 
−1, if ( yin ) j  0.
• THIS TYPE OF NETWORK IS USED IN SPEECH PROCESSING,
IMAGE PROCESSING, PATTERN CLASSIFICATION ETC.
 EXAMPLE---AUTOASSOCIATIVE NETWORK

• Train the autoassociative network for input vector [-1 1 1 1]

and also test the network for the same input vector.
• Test the autoassociative network with one missing, one
mistake, two missing and two mistake entries in test vector.

• The weight matrix W is computed from the formula

P
W =  sT ( p).s ( p )
p =1

• Here, P = 1.
 COMPUTATIONS
• S

 −1  1 −1 −1 −1
1  −1 1 1 1 
W =   . −1 1 1 1 =  
1  −1 1 1 1 
   
 
1  −1 1 1 1 
• Case-1: testing the network with the same input vector
• Test input: [-1 1 1 1]
• The weight obtained above is used as the initial weights
 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 111].   = [−4 4 4 4]
 −1 1 1 1
 
 −1 1 1 1 
 COMPUTATIONS

• Applying the activation function

1, if ( yin ) j  0;

y j = f (( yin ) j ) = 
−1, if ( yin ) j  0.

• Over net input, we get
y = [-1 1 1 1]
• Hence, the correct response is obtained
• TESTING AGAINST ONE MISSING ENTRY
• Case 1: [0 1 1 1] (First Component is missing)
• Compute the net input
 COMPUTATIONS
• S

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [0 111].   = [ −3 3 3 3]
 −1 1 1 1
 
 −1 1 1 1 

• Applying the activation function taken above, we get

• y = [-1 1 1 1]. So, the response is correct
• TESTING AGAINST ONE MISSING ENTRY
• Case 2: [-1 1 0 1] (Third Component is Missing)
• Compute the net input
 COMPUTATIONS
• S

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 1 0 1].   = [ −3 3 3 3]
 −1 1 1 1
 
 −1 1 1 1 
• Applying the activation function taken above, we get
• y = [ -1 1 1 1]
• The response is correct.
• WE CAN TEST FOR OTHER MISSING ENTRIES SIMILARLY
• Testing the network against one mistake entry
• Case 1: Let the input be [-1 -1 1 1] (Second Entry is a Mistake)
 COMPUTATIONS

• Compute the net input

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 − 111].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1 

• Applying the same activation function taken above,

• y = [-1 1 1 1]
• So, the response is correct
• Case 2: Let the input be [1 1 1 1] (First entry is mistaken)
 COMPUTATIONS

• Compute the net input

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [1 111].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1 
• Applying the activation function taken above
• y = [-1 1 1 1]
• So, the response is correct
• Testing the net against two missing entries
• Case 1: Let the input be [0 0 1 1] (First two entries are
missing)
 COMPUTATIONS

• Compute the net input

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [0 0 11].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1 

• Applying the activation function taken above

• y = [-1 1 1 1], which is the correct response
• Case 2: Let the input be [-1 0 0 1] (Second & Third inputs
are missing)
• Compute the net input
 COMPUTATIONS
• S

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 0 0 1].   = [−2 2 2 2]
 −1 1 1 1
 
 −1 1 1 1

• Applying the input function taken above

• y = [-1 1 1 1], which is the correct response
• Testing the net against two mistaken entries
• Let the input be [-1 -1 -1 1] (Second & Third inputs are
missing)
 COMPUTATIONS

• Compute the net input

 1 −1 −1 −1
 −1 1 1 1 
( yin ) j = x. W = [−1 − 1 − 11].   = [0 0 0 0]
 −1 1 1 1
 
 −1 1 1 1
• Applying the activation function taken above, we get
• y = [0 0 0 0]. Which is not correct.
• SO, THE NETWORK FAILS TO RECOGNISE INPUTS WITH TWO
MISTAKES

• NOTE: WE HAVE TO CHECK FOR ALL POSSIBLE INPUTS UNDER

EACH CASE TO HAVE A POSITIVE CONCLUSION
 HETEROASSOCIATIVE MEMORY NETWORK

• The training input and target output are different

• The weights are determined in such a way that the net can
store a set of ‘P’ pattern associations
• Each input vector s(p) has ‘n’ components and each output
vector t(p) has ‘m’ components (p = 1, 2, ….P)
• Determination of weights: Hebb Rule or Delta Rule is used

• THE NET FINDS AN APPROPRIATE OUTPUT VECTOR

CORRESPONDING TO AN INPUT VECTOR, WHICH MAY BE
ONE OF THE STORED PATTERNS OR A NEW PATTERN
 ARCHITECTURE OF HETEROASSOCIATIVE
MEMORY NETWORK
• A

x1 w11 y1
X1 Y1
wi1
wn1

w1i
xi yi
Xi wii Yi
wni

w1m
wim

xn ym
Xn wnm Ym

Differs from Auto- associative network in the number of output units

TESTING ALGORITHM FOR HETEROASSOCIATIVE
MEMORY NETWORK
• STEP 0: Initialize the weights from the training algorithm
• STEP 1: Perform steps 2 – 4 for each input vector presented
• STEP 2: Set the activation for the input layer units equal to
that of the current input vector given, xi
• STEP 3: Calculate the net input to the output units using
n
( yin ) j =  xi .wij , j = 1, 2,...m
i =1

• STEP 4: Determine the activations for the output units as:

1, if ( yin ) j  0;

y j = 0, if ( yin ) j = 0; Differs from auto-associative here

 −1, if ( yin ) j  0.
 HETEROASSOCIATIVE MEMORY NETWORK
CONTD…
• There exist weighted interconnections between the input and
output layers
• The input and output layers are not correlated with each
other
• We can also use the following binary activation function

1, if ( yin ) j  0;

yj = 
−1, if ( yin ) j  0.

 EXAMPLE-HETEROASSOCIATIVE MEMORY
NETWORKS
• Train a heteroassociative memory network using Hebb rule
to store input row vector s = ( s1 , s2 , s3 , s4 ) to the output
vector t = (t1 , t2 ) as given in the table below:
•
Input targets s1 s2 s3 s4 t1 t2
1st 1 0 1 0 1 0
2nd 1 0 0 1 1 0
3rd 1 1 0 0 0 1
4th 0 0 1 1 0 1
 THE NEURAL NET
• s

x1
X1 w11
y1
w12 Y1
x2 w21
X2 w22

w31
x3 w32 y2
X3 Y2
w41
x4 w42
X4
 COMPUTATIONS

• We use Hebb rule to determine the weights

• The initial weights are all zeros

• For the first pair

x1 = 1, x2 = 0, x3 = 1, x4 = 0, y1 = 1, y2 = 0
• Set the input and output pairs

• Weight Updation: Formula

• wij (new) = wij (old) + x i . y j
 UPDATED WEIGHTS

w11 (new) = w11 (old) + x1 . y1 = 0 + 1 1 = 1

w12 (new) = w12 (old) + x1 . y2 = 0 + 0  1 = 0
w21 (new) = w21 (old) + x 2 . y1 = 0 + 1 1 = 1
w22 (new) = w22 (old) + x 2 . y2 = 0 + 1 1 = 0
w31 (new) = w31 (old) + x 3 . y1 = 0 + 1 0 = 0
w32 (new) = w32 (old) + x 3 . y2 = 0 + 0  0 = 0
w41 (new) = w41 (old) + x 4 . y1 = 0 + 1 0 = 0
w42 (new) = w42 (old) + x 4 . y2 = 0 + 0  0 = 0
 UPDATED WEIGHTS
• 2

𝑤11 (𝑛𝑒𝑤) = 𝑤11 (𝑜𝑙𝑑) + 𝑥1 . 𝑦1 = 0 + 1 × 1 = 2

𝑤12 (𝑛𝑒𝑤) = 𝑤12 (𝑜𝑙𝑑) + 𝑥1 . 𝑦2 = 0 + 0 × 1 = 0
𝑤21 (𝑛𝑒𝑤) = 𝑤21 (𝑜𝑙𝑑) + 𝑥2 . 𝑦1 = 0 + 1 × 1 = 0
𝑤22 (𝑛𝑒𝑤) = 𝑤22 (𝑜𝑙𝑑) + 𝑥2 . 𝑦2 = 0 + 1 × 1 = 0
𝑤31 (𝑛𝑒𝑤) = 𝑤31 (𝑜𝑙𝑑) + 𝑥3 . 𝑦1 = 0 + 1 × 0 = 1
𝑤32 (𝑛𝑒𝑤) = 𝑤32 (𝑜𝑙𝑑) + 𝑥3 . 𝑦2 = 0 + 0 × 0 = 0
𝑤41 (𝑛𝑒𝑤) = 𝑤41 (𝑜𝑙𝑑) + 𝑥4 . 𝑦1 = 0 + 1 × 0 = 1
𝑤42 (𝑛𝑒𝑤) = 𝑤42 (𝑜𝑙𝑑) + 𝑥4 . 𝑦2 = 0 + 0 × 0 = 0
 COMPUTATIONS

• For the third pair

• The initial weights are the outputs from the above updation
• The input-output vector pair is
x1 = 1, x2 = 1, x3 = 0, x4 = 0, y1 = 0, y2 = 1
• Set the input and output pairs
• The weight updation formula is same as above
• The weights change in only two out of 8 cases:
w12 (new) = w12 (old) + x1 . y2 = 0 + 1 1 = 1
w22 (new) = w22 (old) + x 2 . y2 = 0 + 1 1 = 1
 UPDATED WEIGHTS
• 3

𝑤11 (𝑛𝑒𝑤) = 𝑤11 (𝑜𝑙𝑑) + 𝑥1 . 𝑦1 = 0 + 1 × 1 = 2

𝑤12 (𝑛𝑒𝑤) = 𝑤12 (𝑜𝑙𝑑) + 𝑥1 . 𝑦2 = 0 + 0 × 1 = 1
𝑤21 (𝑛𝑒𝑤) = 𝑤21 (𝑜𝑙𝑑) + 𝑥2 . 𝑦1 = 0 + 1 × 1 = 0
𝑤22 (𝑛𝑒𝑤) = 𝑤22 (𝑜𝑙𝑑) + 𝑥2 . 𝑦2 = 0 + 1 × 1 = 1
𝑤31 (𝑛𝑒𝑤) = 𝑤31 (𝑜𝑙𝑑) + 𝑥3 . 𝑦1 = 0 + 1 × 0 = 1
𝑤32 (𝑛𝑒𝑤) = 𝑤32 (𝑜𝑙𝑑) + 𝑥3 . 𝑦2 = 0 + 0 × 0 = 0
𝑤41 (𝑛𝑒𝑤) = 𝑤41 (𝑜𝑙𝑑) + 𝑥4 . 𝑦1 = 0 + 1 × 0 = 1
𝑤42 (𝑛𝑒𝑤) = 𝑤42 (𝑜𝑙𝑑) + 𝑥4 . 𝑦2 = 0 + 0 × 0 = 0
 COMPUTATIONS

• For the fourth pair

• The initial weights are the outputs from the above updation
• The input-output vector pair is
x1 = 0, x2 = 0, x3 = 1, x4 = 1, y1 = 0, y2 = 1
• Set the input and output pairs
• The weight updation formula is same as above
• The weights change in only two of the 8 cases:
w32 (new) = w32 (old) + x1 . y2 = 0 + 1 1 = 1
w42 (new) = w42 (old) + x 4 . y2 = 0 + 1 1 = 1
 COMPUTATIONS

• The final weights after all the input/output vectors are used
are
 w11 w12   2 1
w w22   0 1
W =  21 =
 w31 w32  1 1
   
 w41 w42  1 1
 BIDIRECTIONAL ASSOCIATIVE MEMORY

• It was developed by Kosko in 1988

• It performs both forward and backward searches
• It uses Hebb rule
• It associates patterns from set A to set B and vice versa
• It responds for input from both layers
• It is a hetero-associative pattern matching network that
encodes binary or bipolar patterns using Hebbian learning
rule
 TYPES OF BAM

• There are two types of BAMs

➢ Discrete
➢ Continuous
• The characterization depends upon the activation functions
used
• Otherwise, the architecture is same
• Discrete BAM:
• Binary or Bipolar activation functions are used
• Continuous BAM:
• Binary sigmoid or Bipolar sigmoid functions are used
 THE BAM ARCHITECTURE
• A

Layer X W Layer Y
x1 w11 y1
X1 Y1
wi1
wn1

w1i
xi yi
Xi wii Yi
wni

w1m
wim

xn ym
Xn wnm Ym
WT
 BAM ARCHITECTURE

• Consists of two layers of neurons

• These layers are connected by directed weighted path
interconnections

• The network dynamics involves two layers of interaction

• The signals are sent back and forth between the two layers
until all neurons reach equilibrium

• The weighs associated are bidirectional

• It can respond to inputs in either layer
 BAM ARCHITECTURE CONTD…

• The weight matrix from one direction (say A to B) is W

• The weight matrix from the other direction (B to A) is W T

• A definite non-zero threshold is assigned

• The activation function is a step function

• When compared to binary vectors the bipolar vectors

improve the performance of the net to a large extent
 THE WEIGHT MATRICES
 w11 . w1 j . w1m 
 . . . . . 

W =  wi1 . wij . wim 
 
 . . . . . 
 wn1 . wnj . wnm 


 w11 . wi1 . wn1   w11 . w1i . w1n 

 . . . . .   . . . . . 

W =  w1 j
T
. wij . wnj  =  w j1 . w ji . w jn 
   
 . . . . .   . . . . . 
 w1m . wim . wnm   wm1 . wmi . wmn 
 DISCRETE BIDIRECTIONAL ASSOCIATIVE
MEMORY
• The diagram is as shown earlier
• When the memory neurons are being activated by putting an
initial vector at the input of a layer, the network evolves a two-
pattern stable state
• Each pattern at the output of one layer
• The network involves two layers of interaction between each
other
• The two bivalent forms (Binary and Bipolar) of BAM are related
to each other
• The weights in both the cases are found as the sum of the
outer products of the bipolar form of the given training vector
pairs
 DETERMINATION OF WEIGHTS

• Let the ‘P’ number of input/target pairs of vectors be denoted

by (s(p), t(p)), p = 1,2,…P
• Then the weight matrix to store a set of input/target vectors
P=1, 2,…P
s ( p ) = ( s1 ( p ),..., si ( p ),..., sn ( p ))
t(p) = (t1 ( p ),..., t j ( p ),..., t m ( p ))

• can be determined by Hebb rule for determining weights for

associative networks P

• In case of binary input vectors wij =  [2si ( p) − 1].[2t j ( p) − 1]

p =1
P
• In case of bipolar input vectors wij =  siT ( p).t j ( p)
p =1
 ACTIVATION FUNCTIONS FOR BAM

• The step activation function with nonzero threshold is used

as the activation function for discrete BAM network
• The activation function is based on whether the input vector
pairs (at the input and target vector) are binary or bipolar
• ACTIVATION FUNCTION FOR ‘Y’ LAYER
• 1. If both the layers have binary input vectors
1, if ( yin ) j  0;

y j =  y j , if ( yin ) j = 0;

0, if ( yin ) j  0.
 ACTIVATION FUNCTIONS FOR BAM

2. When the input vector is bipolar

1, if ( yin ) j   j ;

y j =  y j , if ( yin ) j =  j ;

−1, if ( yin ) j   j .
• INPUT FUNCTION FOR ‘X’ LAYER
1. When the input vector is binary
1, if (x in )i  0;

xi =  xi , if (x in )i = 0;
0, if (x )  0.
 in i
 ACTIVATION FUNCTIONS FOR BAM

2. When the input vector is bipolar

1, if (x in )i  i ;

xi =  xi , if (x in )i = i ;
−1, if (x )   .
 in i i
 TESTING ALGORITHM FOR DISCRETE BAM

• This algorithm is used to test the noisy patterns entering into

the network
• Basing upon the training algorithm, weights are determined
• Using the weights the net input is calculated for the given test
pattern
• Activations are applied over it to recognize the test patterns
• ALGORITHM:
• STEP 0: Initialize the weights to store ‘p’ vectors
• STEP 1: Perform Steps 2 to 6 for each testing input
• STEP 2: (i) Set the activations of the X layer to the current
input pattern, that is presenting the input pattern x to X.
 TESTING ALGORITHM FOR DISCRETE BAM
CONTD…
• (ii) Set the activations of the ‘Y’ layer similarly by presenting
the input pattern ‘y’.
• (iii) Though it is bidirectional memory, at one time step,
signals can be sent from only one layer. So, one of the input
patterns may be zero vector
• STEP 3: Perform steps 4 to 6 when the activations are not
converging
• STEP 4: Update the activations of units in Y layer. Calculate the
net input by n
( yin ) j =  xi .wij
i =1
 TESTING ALGORITHM FOR DISCRETE BAM
CONTD…
• Applying activations, we obtain y j = f (( yin ) j )
• Send this signal to the X layer
• STEP 5: Update the activations of units in X layer
• Calculate the net input m
( xin )i =  y j .wij
j =1
• Apply the activations over the net input xi = f (( xin )i )
• Send this signal to the Y layer
• STEP 6: Test for convergence of the net. The convergence
occurs if the activation vectors x and y reach equilibrium.
• If this occurs stop, otherwise continue.
 CONTINUOUS BAM

• It transforms the input smoothly and continuously in the

range 0-1 using logistic sigmoid functions as the activation
functions for all units
• This activation function may be binary sigmoidal function or
it may be bipolar sigmoidal function
• When a bipolar sigmoidal function with a high gain is chosen,
the continuous BAM might converge to a state of vectors
which will approach to the vertices of a cube
• In such a circumstance it acts like a discrete BAM
 CONTINUOUS BAM CONTD…

• In case of binary inputs, (s(p), t(p)), p = 1,2,…P; the weight is

determined by the formula
P
wij =  [2si ( p) − 1].[2t j ( p) − 1]
p =1

• The activation function is the logistic sigmoidal function

• If it is binary logistic function then the activation function is
given by
1
f (( yin ) j ) = −( y )
1 + e in j
 CONTINUOUS BAM CONTD…

• If the activation function used is a bipolar function then it is

given by
−( y )
2 1 − e in j
f (( yin ) j ) = − ( yin ) j
−1 = −( y )
1+ e 1 + e in j

• These activations are applied over the net input to calculate

the output. The net input can be calculated with a bias as
n
( yin ) j = b j +  xi .wij
i =1
• All these formulae are applicable to the X layer also
 EXAMPLE-5

• Construct and test a BAM network to associate letters E and F

with single bipolar input-output vectors. The target output
for E is [-1, 1] and F is [1, 1]. The display matrix size is 5 x 3.
The input patterns are:
     
 (-1)   
   (1) 
 
   
• Target out puts are [-1, 1] [1, 1]
 COMPUTATIONS

• The input patterns are

E 1 1 1 1 -1 -1 1 1 1 1 -1 -1 1 1 1
F 1 1 1 1 1 1 1 -1 -1 1 -1 -1 1 -1 -1

• The output and weights are

E -1 1 W1
F 1 1 W2
 COMPUTATIONS

• Since we are considering bipolar input and outputs, the

weight matrix is computed by using the formula:

W =  sT ( p ). t ( p )

• There are two weight components W1 and W2

W1 = [1 111 − 1 − 11111 − 1 − 1111]T [ −11]

•
W2 = [1 111111 − 1 − 11 − 1 − 11 − 1 − 1]T [11]
 COMPUTATIONS
 −1 1 1 1
• So,  −1 1 1 
 1  
 −1 1 1 1
   
 −1 1
1 1
1 1 1
−1  
 
1 −1 1 1
 −1 1 1
1  
 
• W1 =  −1 1 and W2 =  −1 −1
   
 −1 1  −1 −1
 −1 1 1 1
   
1 −1  −1 −1
1  −1 −1
−1  
 
 −1 1 1 1
 −1  −1 −1
1  
 
 −1 1   −1 −1
 COMPUTATIONS
0 2
0 2
• The total weight matrix is  
0 2
 
 0 2 
2 0
 
2 0
0 2
 
W = W1 + W2 =  −2 0 
 
 − 2 0 
 0 −2 
 
 0 −2 
 0 −2 
 
0 2
 −2 0 
 
 −2 0 
 TESTING THE NETWORK

• We test the network with test vectors E and F

• Test pattern E 0 2
0 2 

0 2
 
0 2
2 0
 
 −2 0
0 2
 
yin = [1111 − 1 − 11111 − 1 − 1111]  −2 0  = [ −12 18]
 
 −2 0
0 2
 
0 −2 
• Applying the activations, we get 0 −2 
 
• Y = [-1 1], which is the correct response 0 2
 −2 0
 
 −2 0 
 TESTING THE NETWORK
0 2
• Test pattern F 0 2
 
0 2
 
 0 2 
2 0
 
 −2 0 
0 2
 
yin = [1111111 − 1 − 11 − 1 − 11 − 1 − 1]  −2 0  = [12 18]
 
 −2 0 
0 2
 
 0 −2 
• Applying activations over the net input  0 −2 
 
we get y = [1 1], which is correct 0 2
 −2 0 
 
 −2 0 
 BACKWARD MOVEMENT

• Y vector is taken as the input

• The weight matrix here is the transpose of the original

• So,

 0 0 0 0 2 2 0 −2 −2 0 0 0 0 −2 −2 
T
=
0 
W
2 2 2 2 0 0 2 0 0 2 −2 −2 2 0
 TESTING THE NETWORK

• For the test pattern E, the net input is [-1 1]

yin = x. W T
0 0 0 0 2 2 0 −2 −2 0 0 0 0 −2 −2 
= [−11]  
 2 2 2 2 0 0 2 0 0 2 −2 −2 2 0 0 
yin =  2 2 2 2 −2 −2 2 2 2 2 −2 −2 2 2 2
• Applying the activation functions, we get
y = 1 1 1 1 −1 −1 1 1 1 1 −1 −1 1 1 1

• Which is the correct response

 TESTING THE NETWORK

• For the pattern F:

• Here, the input is [1 1]
yin = x. W T
 0 0 0 0 2 2 0 −2 −2 0 0 0 0 −2 −2 
= [11] 
2 2 2 2 0 0 2 0 0 2 −2 −2 2 0 0 

yin =  2 2 2 2 2 2 2 −2 −2 2 −2 −2 2 −2 −2 

• Applying the activation function we get

y = 1 1 1 1 1 1 1 −1 −1 1 −1 −1 1 −1 −1
• This is the correct response
HOPFIELD NETWORK
 HOPFIELD NETWORK

• Introduced by J. J. Hopfield in the year 1982

• It has got many useful applications in associative memory
and various optimisation problems
• There are two types of Hopfield networks:
• Discrete Hopfield networks
• Continuous Hopfield networks
• Biological neurons are asynchronous (not existing or
occurring at the same time ) by nature
• Hopfield networks is developed to confirm this nature
• Only one unit updates its activation at a time
 HOPFIELD NETWORK CONTD…

• Each unit is found to continuously receive an external signal

along with the signals it receives from the other units in the
net
• When a single layer recurrent network is performing a
sequential updating process, an input pattern is first applied
to the network
• The network output is found to be initialised accordingly
• After that the input pattern is removed and the output that is
initialised becomes the new updated input through the
feedback connections
• The first updated input forces the first updated output, which
in turn acts as the second updated input through the
feedback and results in second updated output
 HOPFIELD NETWORK CONTD…

• This transition process continues until no new updated

responses are produced and the network reaches its
equilibrium
 DISCRETE HOPFIELD NETWORK

• It is network which is:

• Auto-associative
• Fully interconnected
• Single layer
• Feed back
• It is called as recurrent network
• In it each processing element has two outputs
• One output is inverting and the other one non-inverting
• The output from each processing element is fed back to the
input of other processing elements but not to itself
 DISCRETE HOPFIELD NETWORK

• It takes two-valued inputs: {0, 1} or {-1, +1}

• The use of bipolar inputs makes the analysis easier
• The net uses symmetrical weights with no self-connections
• That is 𝑤𝑖𝑗 = 𝑤𝑗𝑖 and 𝑤𝑖𝑖 = 0, ∀𝑖
• Key Points:
• Only one unit updates its activation at a time
• Each unit is found to continuously receive an external signal
along with the signals it receives from the other units in the
net
• When a single-layer recurrent network is performing a
sequential updating process, an input pattern is first applied
to the network and the output is initialized accordingly
 ARCHITECTURE OF A DISCRETE HOPFIELD NET
• g
x1 x2 xi xn

wi1 win wn1

w21 w2n
w2i wi2 wn2
w1n
w1i wni
w12

Y1 Y2 Yi Yn

y1 y2 yi yn
 TRAINING ALGORITHM FOR DISCRETE
HOPFIELD NETWORK
• Hopfield’s first description used binary input vectors and only
later on bipolar input vectors were used
• For storing a set of binary patterns s(p), p = 1, 2,…P, where
s(p) = 𝑠1 (𝑝), 𝑠2 (𝑝), . . . 𝑠𝑛 (𝑝) , the weight matrix W is given
by
𝑃
• 𝑤𝑖𝑗 = ෍ 2𝑠𝑖 (𝑝) − 1][2𝑠𝑗 (𝑝) − 1 , for 𝑖 ≠ 𝑗
𝑝=1
• For storing a set of bipolar inputs the weight matrix W is given
by
𝑃
• 𝑤𝑖𝑗 = ෍ 𝑠𝑖 (𝑝). 𝑠𝑗 (𝑝൯ , for 𝑖 ≠ 𝑗
𝑝=1
 TRAINING ALGORITHM FOR DISCRETE
HOPFIELD NETWORK
• STEP 0: Initialize the weights to store patterns, i. e. weights
obtained from training algorithm using Hebb rule
• STEP 1: When the activations of the net are not converged,
perform steps 2 - 8
• STEP 2: Perform steps 3 -7 for each input vector X
• STEP 3: Make the initial activation of the net equal to the
external input vector X (i. e. 𝑦𝑖 = 𝑥𝑖 (𝑖 = 1,2. . . 𝑛)
• STEP 4: Perform steps 5 – 7 for each unit 𝑌𝑖
• STEP 5: Calculate the net input of the network:

• 𝑦𝑖𝑛 𝑖 = 𝑥𝑖 + ෍ 𝑦𝑗 𝑤𝑗𝑖
𝑗
 TRAINING ALGORITHM FOR DISCRETE
HOPFIELD NETWORK
• STEP 6: Apply the activations over the net input to calculate
the output:
1, if ( yin )i  i ;

yi =  yi , if ( yin )i = i ;
0, if ( y )   .
 in i i

• where 𝜃𝑖 is the threshold and is normally taken as zero

• STEP 7: Now feed back the obtained output 𝑦𝑖 to all other
units. Thus the activation vectors are updated
• STEP 8: Finally test the network for convergence
 CONTINUOUS HOPFIELD NETWORK

• It is a discrete Hopfield network in which the time is assumed

to be a continuous variable
• The nodes of this network have a continuous graded output
rather than a two-state binary output
• The asynchronous updation of the units allows a function
called as energy function or Liapunov function, for the net
• The existence of this function enables us to prove that the net
will converge to a stable set of activations
• The energy of the network decreases continuously with time