Basics of Image Processing

Prof. Iyad Jafar

OUTLINE
• Image Statistics

• Noise in Images

• Point Processing

• Neighborhood Processing and Filters

• Template Matching

• Image Scaling and Interpolation

• Geometric Transformations 2
IMAGE STATISTICS

3
IMAGE STATISTICS
• An image is an array of numbers, calculate some statistics.

Minimum 2
Maximum 255
Mean (μ) 86
Median 59
Standard Deviation (σ) 69

4
IMAGE STATISTICS

5
IMAGE HISTOGRAM
• Image histogram shows the distribution of pixels intensities in the image

h ( 𝑘 ) = 𝑛𝑘
𝑛𝑘
h 𝑛𝑜𝑟𝑚 ( 𝑘 )=
𝑀×𝑁

6
IMAGE HISTOGRAM

7
NOISE IN IMAGES

8
NOISE
• Unwanted random variation of brightness or color information
in images
• Noise is introduced into a digital image during image
• Acquisition
• quality of sensing elements and the environmental conditions
• Transmission
• channel interference

• Usually it is assumed to be additive and uncorrelated with the

pixels values and their spatial coordinates
𝑔 ( 𝑥 , 𝑦 )= 𝑓 ( 𝑥 , 𝑦 )+ 𝑛( 𝑥 , 𝑦 )
9
NOISE MODELS
• Gaussian Noise

2
(𝑥 −𝜇)
−
1 2𝜎
2

𝑓 (𝑥 )= 𝑒
𝜎 √2 𝜋

10
NOISE MODELS
• Impulse (salt &
pepper) Noise

11
POINT PROCESSING

12
POINT PROCESSING
• Point processing refers to modifying individual pixels values
using some transformation
𝑔 ( 𝑥 , 𝑦 )=𝑇 [ 𝑓 ( 𝑥 , 𝑦 ) ]
• Transformation depends on the pixel value only!
• Transformation could be
• Add, subtract, multiply and divide an image with a constant
• T could be any mathematical function (log, linear, sine,….)
• Add, subtract, multiply and divide two images of the same size
• Thresholding
• ….
13
ADDING AND SUBTRACTING CONSTANTS
• We can perform all operations as long as they are meaningful.
• Clip out of range values after operation.

14
LINEAR SCALING
(STRETCHING/COMPRESSION)

𝐺 ( 𝑥 , 𝑦 )=𝑇 (𝑂 ( 𝑥 , 𝑦 ) )

15
IMAGE NEGATIVE

𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒=255 − 𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙
16
ARITHMETIC OPERATIONS BETWEEN ARRAYS
Operations are performed array-wise, i.e. element
by element
Images should be of the same size

17
ARITHMETIC OPERATIONS BETWEEN ARRAYS
• Averaging multiple images captured for the same scene reduces
uncorrelated zero-mean noise
K=5 K=10`

K=10 K=20 K=100

18
ARITHMETIC OPERATIONS BETWEEN ARRAYS
• Subtraction in mask-model radiography

Image before Image after

Iodine Iodine
injection injection
f(x,y) (mask h(x,y))

Difference Enhanced
image difference
g(x,y) image

19
ARITHMETIC OPERATIONS BETWEEN ARRAYS
• Region of interest (ROI) extraction by multiplying with a mask

Whole Mask ROI

image

20
INTENSITY SLICING AND THRESHOLDING
• Highlight a range of intensities/colors

𝑔 ( 𝑥 , 𝑦 )=
{ 0
𝑓 (𝑥 , 𝑦)
, 30 ≤ 𝑓 ( 𝑥 , 𝑦 ) ≤100
, 𝑜𝑡 h𝑒𝑟𝑤𝑖𝑠𝑒
𝑔 ( 𝑥 , 𝑦 )= {0
255
, 30 ≤ 𝑓 ( 𝑥 , 𝑦 ) ≤ 100
, 𝑜𝑡 h 𝑒𝑟𝑤𝑖𝑠𝑒
21
BITWISE LOGICAL OPERATIONS ON BINARY
IMAGES

22
MANIPULATING COLOR

23
MANIPULATING COLOR
• Color slicing/segmentation

{
• For each color channel ri
( )
𝑛

𝑆 𝑖=
128 , ∑ 𝑖 𝑗 > 𝑅 20
( 𝑟 − 𝑎 )
2

• a is the center of sphere 𝑗 =1

𝑟𝑖 , 𝑂𝑡h𝑒𝑟𝑤𝑖𝑠𝑒
• R0 is the radius
24
NEIGHBORHOOD PROCESSING

25
NEIGHBORHOOD
• A pixel at coordinates has two horizontal and two vertical
x
neighbors with coordinates
(x,y)

• 4-neighbors of p
y
• The four diagonal neighbors of p have coordinates

• All together are called 8-neighbors of p

• Neighborhood size could be larger, usually with odd dimensions
26
NEIGHBORHOOD PROCESSING
• The pixel value is transformed based
on the values in its neighborhood
• The process is repeated for all pixel in
the image
• Examples
• Minimum
• Maximum
• Median
• Average
• Mode
•… 27
NEIGHBORHOOD PROCESSING - MEDIAN
• Replace the pixel with the median of its neighborhood

x
10 10 50 10 10 10 50 10

20 30 10 15 20 20 15 15

20 150 10 40 20 20 20 40

30 20 20 20 30 30 30 20

40 40 30 30 40 40 30 30

y 28
NEIGHBORHOOD PROCESSING - MEDIAN
• Median operation has the
effect of blurring/smoothing
the image

29
NEIGHBORHOOD PROCESSING - MEDIAN
• Median filtering can be
useful in reducing the
noise

30
NEIGHBORHOOD PROCESSING - AVERAGE
• Replacing the pixel with the average of its neighborhood

x
10 10 50 10 10 10 50 10

20 30 10 15 20 34 36 15

20 150 10 40 20 34 35 40

30 20 20 20 30 40 40 20

40 40 30 30 40 40 30 30

31
y
NEIGHBORHOOD PROCESSING - AVERAGE
• This can be represented as element-wise multiplication between the
neighborhood and a 3x3 array/mask/filter/kernel then summing up the
values
f(x-1,y-1) f(x-1,y) f(x-1,y+1) 1/9 1/9 1/9

f(x,y-1) f(x,y) f(x,y+1)

x 1/9 1/9 1/9
 g(x,y)

f(x+1,y-1) f(x+1,y) f(x+1,y+1) 1/9 1/9 1/9

3x3 Neighbourhood around 3x3 Filter/mask/kernel Value in output image

pixel at location (x,y) in the at location (x,y)
original Image Pixels 32
CORRELATION
• Generally, the kernel/mask/filter w may have any weights
y
w(-1,-1) w(-1,0) w(-1,1) f(x-1,y-1) f(x-1,y) f(x-1,y+1)

w(0,-1) w(0,0) w(0,1)

x f(x,y-1) f(x,y) f(x,y+1)

3x3 Kernel w(1,-1) w(1,0) w(1,1) f(x+1,y-1) f(x+1,y) f(x+1,y+1)

x
Image f (x, y) Filter/mask/kernel Original Image Pixels

a b
g( x, y )    w( s,t ) f ( x  s, y  t )
s  a t  b

• This operation is called correlation! 33
CONVOLUTION
• A strongly related operation to correlation is convolution
• Convolution is similar to correlation, except that the filter mask
is flipped by 180o around its center before sliding
1 2 3 𝑎 𝑏
7 1 6 𝑔 ( 𝑥 , 𝑦 )= 𝑓 ⋆ 𝑤= ∑ ∑ 𝑤 ( 𝑠, 𝑡 ) 𝑓 (𝑥+𝑠 , 𝑦 +𝑡)
4 5 0 𝑠=− 𝑎 𝑡 =−𝑏

Kernel for Correlation

0 5 4 𝑎 𝑏
6 1 7 𝑔 ( 𝑥 , 𝑦 )= 𝑓 ∗ 𝑤= ∑ ∑ 𝑤 ( 𝑠, 𝑡 ) 𝑓 (𝑥 − 𝑠, 𝑦 −𝑡)
3 2 1 𝑠=− 𝑎 𝑡=− 𝑏

Kernel for Convolution 34

CORRELATION AND CONVOLUTION
• Convolution and correlation are calculated
from the first to last overlap W
0
W
1
W
2

W W W
3 4 5

W W
5
W
1 1 5 1
• Border pixels? 6 7 8

6 1 6 6 3
• Padding m-1 and n-1 from all sides 7 2 1 6 4
• With constant 5 1 3 5 1
• Replicate edge pixels 2 3 1 6 1
• Allow pixels to warp around
• Consider pixels that fall under the mask
• …..

35
CORRELATION AND CONVOLUTION
• Border pixels

36
CORRELATION AND CONVOLUTION
• Size of full correlation output for m×n kernel
and P × Q image
• The size of the output is
5 1 1 5 1
6 1 6 6 3
• Interest in cropped correlation; area, when the 7 2 1 6 4

mask is centered at the first and last pixels 5 1 3 5 1

2 3 1 6 1

37
PROPERTIES OF CONVOLUTION
• Commutative
A⁕B=B⁕A
• Associative
A ⁕(B ⁕ C) = (A ⁕ B) ⁕ C
• Distributes over addition/subtraction
A ⁕ (B+C) = A ⁕ B + A ⁕ C
• Plays well with scalars
x(A ⁕ B) = (xA) ⁕ B = A ⁕(xB)

38
APPLICATIONS OF
CONVOLUTION/CORRELATION
• Much of low-level computer vision is convolutions
• Template Matching
• Smoothing or blurring
• Sharpening
• Derivatives
• Detecting of edges and features
• Super-resolution
• Classification
• …..

39
SMOOTHING FILTER
• A smoothing/averaging/ blurring/box filter
• Reduces noise
• Removes small details
1 1 1 1 1

1 1 1 1 1 1 1 1

1 1 1 X 1/9 1 1 1 1 1 X 1/25
1 1 1
1 1 1 1 1
3x3 Standard averaging box mask
1 1 1 1 1

5x5 Standard averaging box mask

40
SMOOTHING FILTER

41
SMOOTHING FILTER

42
SMOOTHING FILTER

43
GAUSSIAN SMOOTHING FILTER
• Box smoothing filters produce artifacts
• Gaussian filters have smooth changing weights; better
smoothing results

0.06 0.12 0.06

25 5 25

0.12 0.12
0.25
5 5

0.06 0.12 0.06

𝑥
2
2 2
25 5 25
−
1 2𝜎
2
1 −
𝑥 +𝑦
𝑤 ( 𝑥) = 𝑒 𝑤 ( 𝑥 , 𝑦 )= 𝑒 2𝜎
2

√ 2 𝜋 𝜎 2 𝜋𝜎

44
GAUSSIAN SMOOTHING FILTER

45
SHARPENING FILTER
• High response at discontinuities in the image 0 1 0
• edges and noise
1 -4 1
• What is the mathematical expression? 0 1 0
• Can be used for sharpening! Laplacian

46
SHARPENING FILTER
• Try to subtract the filtered image from the
0 -1 0
original image
-1 5 -1
f(x,y) – g(x,y) 0 -1 0

47
TEMPLATE MATCHING

48
TEMPLATE MATCHING USING CORRELATION
• Given: a template (image or image patch)
• Task: Find it in another image

Template (t)

Image (f)
49
TEMPLATE MATCHING USING CORRELATION
• Slide the template over the image and calculate the sum of
squared differences (SSD)!
• Look for minimum value(s)
𝐸 (𝑖, 𝑗)=∑ ∑ ( 𝑓 ( 𝑚,𝑛 ) − 𝑡 ( 𝑚− 𝑖,𝑛− 𝑗 ) )
2
Minimize
𝑚 𝑛

𝐸 ( 𝑖, 𝑗 ) =∑ ∑ 𝑓 ( 𝑚,𝑛 ) +𝑡 ( 𝑚−𝑖,𝑛− 𝑗 ) −2 𝑓 ( 𝑚,𝑛 ) 𝑡 ( 𝑚−𝑖,𝑛− 𝑗 )

2 2

𝑚 𝑛
Maximize

𝑅𝑡𝑓 (𝑖 , 𝑗 )=∑ ∑ 𝑓 (𝑚 ,𝑛 ) 𝑡 ( 𝑚−𝑖,𝑛− 𝑗 ) Cross-correlation

𝑚 𝑛
50
TEMPLATE MATCHING USING CORRELATION
• Problem with cross-correlation!

t f

A B C

𝑅𝑡𝑓 ( 𝐶 ) > 𝑅𝑡𝑓 ( 𝐵 ) > 𝑅𝑡𝑓 ( 𝐴)

We need Rtf(A) to be maximum?

51
TEMPLATE MATCHING USING CORRELATION
• Normalize the cross-correlation to account for differences in
brightness 𝑁 ( 𝑖 , 𝑗 )= ∑ ∑ 𝑓 ( 𝑚 , 𝑛 ) 𝑡 (𝑚 − 𝑖 , 𝑛 − 𝑗 )
𝑚 𝑛

√ ∑ ∑ 𝑓 ( 𝑚 , 𝑛) √ ∑ ∑ 𝑡 ( 𝑚 − 𝑖 , 𝑛 − 𝑗 )
𝑡𝑓 2 2

𝑚 𝑛 𝑚 𝑛

Template (t)

Image (f) Normalized correlation map (Ntf)

52
IMAGE SCALING AND
INTERPOLATION

53
IMAGE DOWNSAMPLING
• Reduce the number of pixels in the image!
• MxN to PxQ
• Zoom out

54
IMAGE DOWNSAMPLING
• Eliminate rows and columns
N
X X X X
X X X X X X X X
Q
X X X X
/2
X X X X X X X X
M P
X X X X
X X X X X X X X
X X X X
X X X X X X X X
55
IMAGE DOWNSAMPLING
• Jaggy/rough edges appear

56
IMAGE DOWNSAMPLING
• Smooth then downsize

57
GAUSSIAN PYRAMIDS

58
IMAGE UPSAMPLING
• Increase the number of pixels in the image!
• MxN to PxQ
• Zoom in

59
IMAGE UPSAMPLING
• Create PxQ empty grid Q
• Distribute known pixels
uniformly
• Fill empty pixels, how?
N
P

60
IMAGE UPSAMPLING
• Nearest neighbor interpolation

• Distance measures
• For pixels p and q with coordinates (x, y) and (u, v)
• Euclidian distance

• City-block distance 61
IMAGE UPSAMPLING
• Bilinear interpolation

• Bicubic interpolation

62
IMAGE UPSAMPLING
GEOMETRIC TRANSFORMATIONS

64
IMAGE TRANSFORMATIONS
• Point and neighborhood processing  Change range of image
𝑇𝒓
𝑔 ( 𝑥 , 𝑦 )=𝑇 𝒓 ( 𝑓 ( 𝑥 , 𝑦 ) )

• Geometric transformation (Image warping)

 change domain (location) of image

𝑇𝒅
𝑔 ( 𝑥 , 𝑦 )=𝑇 𝒅 ( 𝑓 ( 𝑥 , 𝑦 ) )

65
2X2 GEOMETRIC LINEAR TRANSFORMATIONS
• Simplest case is 2D/planner
𝑇𝒅 transformation

𝑝 1=( 𝑥1 , 𝑦 1 ) 𝑝 2=( 𝑥 2 , 𝑦 2)
𝑝 2=𝑇 𝑝 1 • Transformation can be
represented as a 2x2 matrix
[ ] [ ]
𝑥2
𝑦2
=𝑇
𝑥1
𝑦1 (Linear operation)

[ ] [
𝑥2
𝑦2
=
𝑎11
𝑎2 1
𝑎1 2
𝑎22 ][ ]
𝑥1
𝑦1
66
2X2 GEOMETRIC LINEAR TRANSFORMATIONS
• Scaling (same as upscaling)

𝑆
−1
𝑆

Forward Inverse
𝑥 2=𝑎 𝑥 1 𝑥1 =
1
𝑥2
𝑎
𝑦 2=𝑏 𝑦 1 𝑦 1=
1
𝑦2
𝑏

67
FORWARD VS. BACKWARD WARPING
• Forward
• Given T, send the value at
each pixel to the new pixel

• Problems?
• Pixels may not map to an
exact pixel
• Interpolate?!
• Many pixels may map one
pixels
• Information lost
• Holes!

68
FORWARD VS. BACKWARD WARPING
• Backward
• Find out where each pixel
g(x’,y’) should get its value
from
• No exact pixel
• Interpolate from the
neighbors

69
2X2 GEOMETRIC LINEAR TRANSFORMATIONS
• Rotation
𝜃
𝑅
−1
𝑅

Forward Inverse
𝑥 2=𝑥1 𝑐𝑜𝑠 ⁡( 𝜃) − 𝑦 1 𝑠𝑖𝑛( 𝜃) 𝑥1 =𝑥2 𝑐𝑜𝑠 ⁡( 𝜃)+ 𝑦 2 𝑠𝑖𝑛( 𝜃)

𝑦 2=𝑥 1 𝑠𝑖𝑛 ( 𝜃 )+ 𝑦 1 𝑐𝑜𝑠( 𝜃 ) 𝑦 1=− 𝑥 2 𝑠𝑖𝑛 ( 𝜃 ) + 𝑦 2 𝑐𝑜𝑠( 𝜃 )

70
2X2 GEOMETRIC LINEAR TRANSFORMATIONS
• Skew (Shear)

Horizontal Vertical
𝑥 2= 𝑥1 +𝑚 𝑥 𝑦 1 𝑥 2= 𝑥1

𝑦 2= 𝑦 1 𝑦 2=𝑚 𝑦 𝑥1 + 𝑦 1

[ ] [ ][
𝑥2
𝑦2
=𝐾
𝑥1
𝑦1
= 1
𝑚𝑦
0
1 ][ ]
𝑥2
𝑦2
71
2X2 GEOMETRIC LINEAR TRANSFORMATIONS
• Any 2x2 linear transformation
• Origin maps to origin
[ ] [
𝑥2
𝑦2
=
𝑎11
𝑎2 1
𝑎1 2
𝑎22 ][ ]
𝑥1
𝑦1

• Lines map to lines

• Parallel lines remain parallel
• Closed under composition!

𝑝 2=𝑇 1 𝑝1
𝑝 3=𝑇 2 𝑝 2 𝑝 4 =𝑇 3 𝑇 2 𝑇 1 𝑝 1=𝑇 𝑝 1
𝑝 4 =𝑇 3 𝑝3

72
3X3 GEOMETRIC TRANSFORMATIONS
• How about translation? 𝑡𝑥
𝑡𝑦

𝑥 2= 𝑥1 +𝑡 𝑥
𝑦 2= 𝑦 1+ 𝑡 𝑦

• Can we represent in 2x2 matrix T? No

• Homogenous coordinates  add third imaginary dimension
73
3X3 GEOMETRIC TRANSFORMATIONS
𝑡𝑥
𝑡𝑦

𝑥 2= 𝑥1 +𝑡 𝑥
𝑦 2= 𝑦 1+ 𝑡 𝑦

[ ][ ][ ][ ]
𝑥2 ~
𝑥2 1 0 𝑡𝑥 𝑥1
𝑦2 ≡ ~
𝑦2 = 0 1 𝑡𝑦 𝑦1
1 ~
𝑧2 0 0 1 1
74
3X3 GEOMETRIC TRANSFORMATIONS
• We can represent all previous transformations using 3x3
matrices

• Closed under composition!

75
3X3 GEOMETRIC TRANSFORMATIONS
• Transformations of this form

are called affine transformations (linear + translation)

• Properties
• Origin does not map to origin
• Lines map to lines
• Parallel lines remain parallel
• Closed under composition!
76
PROJECTIVE TRANSFORMATIONS
• In general, the transformation of the form

[ ][ ][ ][ ]
𝑥2 ~
𝑥2 h11 h1 2 h1 3 ~
𝑥1
𝑦2 ≡ ~
𝑦 2 = h2 1 h22 h 23 ~
𝑦1
1 ~
𝑧2 𝒉𝟑 𝟏 𝒉 𝟑𝟐 𝒉 𝟑𝟑 ~
𝑧1

represents the projective or perspective transformation and H is

called the projective matrix or homography

77
3X3 GEOMETRIC TRANSFORMATIONS
• The homogenous representation of a 2D
L
point is a 3D point where and it is a ~
𝑧
fictitious such
~
𝑥 ~
𝑦
1 y
𝑥= ~ 𝑦=~
𝑧 𝑧
x

• All points on L are equivalent not equal ~

𝑦

[][ ][]
𝑥 ~
𝑧𝑥 ~
𝑥
𝑝≡ 𝑦 ≡ ~ 𝑦 =~
𝑧𝑦 ≡ ~ 𝑝 ~
𝑥
1 ~
𝑧 ~
𝑧
78
PERSPECTIVE PROJECTION
• Properties
• Origin does not map to origin
• 3D lines project to 2D lines
• Parallel lines don’t remain parallel (converge to a vanishing point)
• Distant objects are smaller
• Close under composition!

79
APPLICATIONS OF HOMOGRAPHY
• Aligning images of the same scene but taken with different
camera locations

80
READINGS
• RG
• 3.1, 3.2, 3.4, 3.5, 3.6, 7.1, 7.2, 7.4, 7.6
• SZ
• 3.1, 3.2, 3.3.1

81
LINKS
• OpenCV Python tutorials
• https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html

• OpenCV Modules
• https://docs.opencv.org/4.x/modules.html

82