Learning from Big Data
Lecture 19: Image recognition and CNNs
Dr. Lloyd T. Elliott, Fall 2022
Why image recognition is difficult
'Typographic attack': pen and paper fool AI into thinking apple is an iPod
• Recognizing objects in real scene
• Variation in lighting and viewpoint
• De nition of objects
• Requires huge amounts of knowledge (even for segmentation and viewport /
lighting)
fi
State of the art
• ADOP: Approximate di erentiable one-pixel point rendering (University of
Erlangen-Nuremberg)
• https://www.youtube.com/watch?v=WJRyu1JUtVw
ff
Things that make it hard
• Segmentation: real scenes are cluttered with other objects:
• Hard to tell which pieces go together
• Parts of an object can be hidden or clipped (occlusion)
• Lighting: Intensities are as much determined by lighting as by nature of object
• Deformation: Wide variety of shapes have the same name
• A ordances: For many objects, function is more important than shape for
de nition
ff
fi
More things that make it hard to recognize objects
• Viewpoint: wide variety of viewpoints for the same object
• "Information hops between input dimensions" dimension hopping
• We don't see this for many types of structured data (medical for example)
Viewpoint invariance
• Each time we look at an object, we have a di erent viewpoint. Unlike other machine
learning tasks
• Humans are so good at viewpoint variation, it's hard to appreciate how di cult it is
• One of the main di culties in computer vision
• Typical approaches:
• Use redundant invariant features
• Bounding boxes
• Replicated features with pooling "convolutional neurons"
ffi
ff
ffi
Invariant feature approach
• Extract large, overlapping / redundant set of features invariant to
transformations (rotation, scaling, translation, shear, stretch)
• Example: centre / surround for visual eld
• Problem: features will overlap with objects that are not in foreground ("parts
of di erent objects")
• Put a box around objects
• Normalize within the box
• Choosing the box is di cult (chicken / egg problem)
ff
ffi
fi
Brute force normalization
• When training recognizer, use well-segmented upright images to t correct
box
• At test time, try all possible boxes in a range of position and scales
• This was used often in computer vision ~2015
fi
Convolutional neural nets
• LeNet 1990s
• Use many di erent copies of the same feature detector with di erent
positions
• A feature detector useful in one place in the
image is likely useful in other areas too
• When we learn, we keep the red arrows all
having the same weights as each other
Red connections all have the same weight
ff
ff
Convolutional neural nets
• Replication greatly reduces the number of free parameters to be learned
• In this example 27 -> 9 weights
• Make many maps, each one with replicates
of the same feature. Di erent maps learn to
detect di erent features.
• Each patch of the image can then be
represented by features of many di erent types
Red connections all have the same weight
ff
ff
ff
Backpropagation with weight constraints
• It's easy to modify the backpropagation algorithm to incorporate linear
constraints between the weights
• We compute gradients as usual, but we modify gradients so that they satisfy
the constraints
w1 = w2 ! w1 = w2
<latexit sha1_base64="LD7YG2MnmpYdYhnbm0b3qUuXbyQ=">AAACFnicbVDLSgMxFM3UV62vUZdugkVwY5kpRd0IRV24rGAf0BmGTJq2oZkHyR1LGfoVbvwVNy4UcSvu/BvTdhBtPXDh5Jx7yb3HjwVXYFlfRm5peWV1Lb9e2Njc2t4xd/caKkokZXUaiUi2fKKY4CGrAwfBWrFkJPAFa/qDq4nfvGdS8Si8g1HM3ID0Qt7llICWPPNk6Nn4Ag+9MnYk7/WBSBkNsXPNBBA8M38eZc8sWiVrCrxI7IwUUYaaZ346nYgmAQuBCqJU27ZicFMigVPBxgUnUSwmdEB6rK1pSAKm3HR61hgfaaWDu5HUFQKeqr8nUhIoNQp83RkQ6Kt5byL+57UT6J67KQ/jBFhIZx91E4EhwpOMcIdLRkGMNCFUcr0rpn0iCQWdZEGHYM+fvEga5ZJ9WqrcVorVyyyOPDpAh+gY2egMVdENqqE6ougBPaEX9Go8Gs/Gm/E+a80Z2cw++gPj4xu9bJ0/</latexit>
• This is done as follows:
• Compute @E @E
<latexit sha1_base64="5xbcRYBlUAx0rL1ZgOhEGzrbHNQ=">AAACKXicfVBbS8MwGE29znmr+uhLcAg+yGjHUB+HIvg4wV1gLSXN0i0sTUuSKqP07/jiX/FFQVFf/SOmW0HdxAOBwznnS/IdP2ZUKst6NxYWl5ZXVktr5fWNza1tc2e3LaNEYNLCEYtE10eSMMpJS1HFSDcWBIU+Ix1/dJH7nVsiJI34jRrHxA3RgNOAYqS05JkNJxAIp06MhKKIwcvsm995dnYM/w3UMs+sWFVrAjhP7IJUQIGmZz47/QgnIeEKMyRlz7Zi5ab5lZiRrOwkksQIj9CA9DTlKCTSTSebZvBQK30YREIfruBE/TmRolDKcejrZIjUUM56ufiX10tUcOamlMeJIhxPHwoSBlUE89pgnwqCFRtrgrCg+q8QD5FuRulyy7oEe3bledKuVe2Tav26XmmcF3WUwD44AEfABqegAa5AE7QABvfgEbyAV+PBeDLejI9pdMEoZvbALxifX8Rbp40=</latexit>
,
@w1 @w2
• Use update @E @E for w1 and w2
<latexit sha1_base64="y+uFVK9ycT8U4mZLwiEDg54HMuc=">AAACPHicfVDNS8MwHE3n15xfU49egkMQhNGOoR6HH+BxovuAtZQ0S7ewtA1J6hilf5gX/whvnrx4UMSrZ7OtoG7ig8DjvfdL8nseZ1Qq03wycguLS8sr+dXC2vrG5lZxe6cpo1hg0sARi0TbQ5IwGpKGooqRNhcEBR4jLW9wPvZbd0RIGoW3asSJE6BeSH2KkdKSW7yxLwhTCA6hzUXEVQRtXyCc2BwJRRGDl+k3H7pWCo/g/5FK6hZLZtmcAM4TKyMlkKHuFh/tboTjgIQKMyRlxzK5cpLxlZiRtGDHknCEB6hHOpqGKCDSSSbLp/BAK13oR0KfUMGJ+nMiQYGUo8DTyQCpvpz1xuJfXidW/qmT0JDHioR4+pAfM6g7GjcJu1QQrNhIE4QF1X+FuI90M0r3XdAlWLMrz5NmpWwdl6vX1VLtLKsjD/bAPjgEFjgBNXAF6qABMLgHz+AVvBkPxovxbnxMozkjm9kFv2B8fgEW766n</latexit>
w/ +
@w1 @w2
• We can thus force backpropagation to use replicated features
What does replicating the features achieve?
• Equivariant activities: the neural activities in the next layer are not invariant to
translation, but they are equivariant
• Representation changes by as much as image
• Invariant knowledge: if a feature can be detected in one location, it can be
detected in other locations too
Pooling the output of replicated feature detectors
• To get invariance in activity, we must pool the output of the convolutional
layer
• Average / maximum together neighbouring replicated detectors to give a
single output to the next level
• Reduces number of inputs to the next layer (means, we can learn more
features)
• Problem: after several levels of this pooling, we lose information about the
precise location of the object (that's ne for kilns for example)
fi
LeNet5
• Yann LeCun and collaborators developed the rst good recognizer for
handwritten digits using backpropagation and feedforward
• Many hidden layers, many maps of replicated units, pooling between layers.
Did not require segmentation
• Was deployed by USPS, ~10% of zip code reading in USA in early 2000s
fi
LeNet5 in tensorflow
• medium.com/@mgazar
model = keras.Sequential()
model.add(layers.Conv2D(filters=6, kernel_size=(3, 3), activation='relu', input_shape=(32,32,1)))
model.add(layers.AveragePooling2D())
model.add(layers.Conv2D(filters=16, kernel_size=(3, 3), activation='relu'))
model.add(layers.AveragePooling2D())
model.add(layers.Flatten())
model.add(layers.Dense(units=120, activation='relu'))
model.add(layers.Dense(units=84, activation='relu'))
model.add(layers.Dense(units=10, activation = 'softmax'))
Prior knowledge in machine learning
• LeNet5 prior knowledge:
• Connectivity
• Weight constraints
• Activation functions
• Less intrusive than hand engineering features, but still pushes the network
towards a particular way of solving the problem
• Alternative: use prior knowledge to create more training data: augment
training data with simulated data (Hofman 1993)
More tricks
• Data augmentation
• Subsample & transform training images (AugMix Hendrycks et al. 2019)
Thank you
