CMPE 403
Fundamentals of Image Processing
Burcu KIR SAVAŞ, PhD.
Image types and color
Course Overview
– Introduction
– What is image processing?
– Image types and color
– Point operations
– Basic Density Conversion Functions
– Histogram Synchronizationg
– Frequency Domain Techniques
Midterm exam
– Wavelets
– Gradients, edges, contours
– Image segmentation
– Image smoothing
image formation
• What determines the brightness of an image pixel?
Light source
properties
Surface
Sensor characteristics shape and orientation
Exposure
Surface reflectance
Optics properties
Slide credit: L. Fei-Fei
digital camera
A digital camera replaces film with a sensor
array
• Each cell in the array is light-sensitive
diode that converts photons to
electrons
• http://electronics.howstuffworks.com/digital-
camera.htm
Slide credit: S. Seitz
digital images
Slide credit: D. Hoiem
digital images
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)
• Image thus represented as a matrix of integer values.
2D
1D
Slide credit: K. Grauman, S. Seitz
image representation
• Digital image: 2D discrete function f
• Pixel:The raster
Smallest image
element (pixel
of an image matrix)
f(x,y)
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
Slide credit: M. J. Black
TODAY
• Perception of color and light
• Color spaces
Why does a visual system need color?
Why does a visual system need
color?
• To tell food and mates.
• To distinguish material changes from shading changes.
• To group parts of one object together in a scene.
• A persons appearance looks normal/healthy.
Slide credit: W. Freeman
What is color?
• Colour, also spelled, Color is the result of interaction
between physical light in the environment and our visual
system
• Color is a psychological property of our visual experiences
when we look at objects and lights,
not a physical property of those objects or lights
(S. Palmer, Vision Science: Photons to Phenomenology)
#thedress
• What is the color of the
dress?
• blue and black
• white and gold
• blue and brown
• What #thedress tell about
our color perception?
11
#thedress
• Let’s take averages
two pieces averages basic pattern
of the dress
12
#thedress
• The dress in the photograph
13
#thedress
• Consider the dress is in shadow.
• Your brain remove the blue cast,
and perceive it as white and gold.
14
#thedress
• The dress in the photograph
15
#thedress
• Consider the dress is in bright light.
• Your brain perceive the dress as a
darker blue and black
16
#thedress
• Answer:
• The dress is actually blue and black.
17
Brightness perception
Consider two pieces of paper, one
black and one white.
Let's say we were outside in bright
sunlight (instead of being in this dingy
lecture hall).
What color do you imagine they would
look like?
Still black and white.
Edward Adelson
Brightness perception
The image of the black paper
outdoors is actually more
intense than the image of white
paper indoors.
Why does the black paper
outdoors still look black even
though it is physically more
intense?
Edward Adelson
Brightness perception
21
Land’s Experiment (1959)
• Cover all patches except a blue rectangle
• Make it look gray by changing illumination
• Uncover the other patches
Color Constancy
We filter out illumination variations
Slide credit: S. Narasimhan
Land’s Experiment (1959)
• Cover all patches except a blue rectangle
• Make it look gray by changing illumination
• Uncover the other patches
Color Constancy
We filter out illumination variations
Slide credit: S. Narasimhan
Color Cube Illusion
Content © 2008 R.Beau Lotto
Color perception
Content © 2008 R.Beau Lotto
Color perception
Content © 2008 R.Beau Lotto
Color perception
Content © 2008 R.Beau Lotto
Image Brightness (Intensity)
Scene
e' (l ) Irradiance
b' q (l )
r ' (l )
Image p ' (l ) Scene
Intensity Reflectance
• Monochromatic Light : (l = li )
b' ( x, y) = r ' ( x, y) e' ( x, y) q(li ) = 1
NOTE: The analysis can be applied to COLORED LIGHT using FILTERS
Slide credit: S. Narasimhan
Color and light
• Color of light arriving at camera depends on
– Spectral reflectance of the surface light is leaving
– Spectral radiance of light falling on that patch
• Color perceived depends on
– Physics of light
– Visual system receptors
– Brain processing, environment
• Color is a phenomenon of human perception;
• it is not a universal property of light
Slide credit: K. Grauman, S. Marschner
Color
White light: composed of
about equal energy in all
wavelengths of the visible
spectrum Colo
r
Newton 1665
Slide credit: B. Freeman, A. Torralba, K. Grauman
Electromagnetic spectrum
• Light is electromagnetic radiation
– exists as oscillations of different frequency (or, wavelength)
Human Luminance Sensitivity Function
Slide credit: A. Efros
The Physics of light
Any source of light can be completely described
physically by its spectrum: the amount of energy emitted
(per time unit) at each wavelength 400 - 700 nm.
Relative
# Photons
spectral
(per power
ms.)
400 500 600 700
Wavelength (nm.)
Slide credit: A. Efros 36
© Stephen E. Palmer, 2002
The Physics of light
Some examples of the spectra of light sources
.
A. Ruby Laser B. Gallium Phosphide Crystal
power
# Photons
# Photons
power
Rel.
Rel.
400 500 600 700 400 500 600 700
Wavelength (nm.) Wavelength (nm.)
C. Tungsten Lightbulb D. Normal Daylight
power
power
# Photons
Photons
#Rel.
Rel.
400 500 600 700 400 500 600 700
Slide credit: A. Efros © Stephen E. Palmer, 2002
The Physics of light
Some examples of the reflectance spectra of surfaces
% Light Reflected
Red Yellow Blue Purple
400 700 400 700 400 700 400 700
Slide credit: A. Efros Wavelength (nm) © Stephen E. Palmer, 2002
Image formation
• What determines the brightness of an image pixel?
Light source
properties
Surface
Sensor characteristics shape and orientation
Exposure
Surface reflectance
Optics properties
Slide credit: L. Fei-Fei
Color mixing
Cartoon spectra for color names:
Credit: W. Freeman
Additive color mixing
Colors combine by adding color spectra
Mixing the three primaries or a secondary with its opposite primary
color produces white light.
Light adds to black.
Credit: W. Freeman
Subtractive color mixing
Colors combine by multiplying color spectra.
Pigments remove color
from incident light
(white).
Credit: W. Freeman
Interaction of light and surfaces
• Reflected color is the
result of interaction of
light source spectrum
with surface
reflectance
Slide credit: A. Efros
Reflection from colored surface
[Stone 2003]
Slide credit: S. Marschner
https://www.dkfindout.com/us/science/light/seeing-color/
The Eye
• Iris - colored annulus with radial muscles
• Pupil - the hole (aperture) whose size is controlled by the iris
• Lens - changes shape by using ciliary muscles (to focus on
objects at different distances)
• Retina - photoreceptor cells
Slide credit: S. Seitz
The eye as a measurement device
• We can model the low-level
behavior of the eye by thinking
of it as a light-measuring machine
– its optics are much like a camera
– its detection mechanism is also
much like a camera
• Light is measured by the
photoreceptors in the retina
– they respond to visible light
– different types respond to different
[Greger et al. 1995]
wavelengths
• The human eye is a camera!
Slide credit: S. Marschner
Layers of the retina
Slide credit: S. Ullman
Eye Movements
• Saccades
– Can be consciously controlled. Related to perceptual
attention.
– 200ms to initiation, 20 to 200ms to carry out. Large
amplitude.
• Microsaccades
– Involuntary. Smaller amplitude. Especially evident during
prolonged fixation. Function debated.
• Smooth pursuit – tracking an object
Slide credit: J. Hays
Receptors Density - Fovea
The fovea itself is the central portion of the macula, which is
responsible for central vision. A large proportion of the
striate cortex is devoted to processing information from the
fovea.
Slide credit: S. Ullman
Human Photoreceptors
Images: Foundations of Vision,
by Brian Wandell, Sinauer Assoc., 1995 Slide Credit: B. Freeman and A. Torralba
Human eye photoreceptor spectral
sensitivities
Images: Foundations of Vision,
by Brian Wandell, Sinauer Assoc., 1995
Slide Credit: B. Freeman and A. Torralba
Two types of light-sensitive receptors
Cones cone-shaped less sensitive
operate in high light color vision
Rods rod-shaped highly sensitive
operate at night gray-scale vision
Rods are responsible for intensity, cones for color perception
Rods and cones are non-uniformly distributed on the retina
Images by Shimon Ullman
Slide credit: A. Efros
Rod / Cone sensitivity
Physiology of Color Vision
.
Three kinds of cones:
RELATIVE ABSORBANCE (%) 440 530 560 nm.
100
S M L
50
400 450 500 550 600 650
WAVELENGTH (nm.)
• Ratio of L to M to S cones: approx. 10:5:1
• Almost no S cones in the center of the fovea
Slide credit: A. Efros © Stephen E. Palmer, 2002
Color perception
M L
Power
Wavelength
Rods and cones act as filters on
the spectrum
Q: How can we represent an
– To get the output of a filter,
entire spectrum with 3
multiply its response curve by
the spectrum, integrate over all numbers?A: We can’t! Most of
wavelengths the information is lost.
• Each cone yields one number
Slide credit: S. Seitz
Digital images
• Sample the 2D space on a regular grid
• Quantize each sample (round to nearest integer)
• Image thus represented as a matrix of integer values.
2D
1D
Slide credit: K. Grauman, S. Seitz
Color Images: Bayer Grid
• Estimate RGB
at G cells from
neighboring values
http://www.cooldictionary.com/
words/Bayer-filter.wikipedia
Slide credit: S. Seitz
Digital color images
Color images, RGB
color space
R G B
Slide credit: K. Grauman
Color spaces: RGB
• Single wavelength primaries
• makes a particular monitor RGB standard
• Good for devices (e.g., phosphors for monitor), but not
for perception
RGB color matching functions
Slide credit: K. Grauman, S. Marschner
Color spaces: RGB
Default color space
0,1,0
R
(G=0,B=0)
1,0,0 G
(R=0,B=0)
0,0,1
Some drawbacks B
(R=0,G=0)
• Strongly correlated channels
• Non-perceptual
Image from: http://en.wikipedia.org/wiki/File:RGB_color_solid_cube.png Slide credit: D. Hoiem
Color spaces: CIE XYZ
• Standardized by CIE (Commission Internationale de
l Eclairage, the standards organization for color science)
• Based on three imaginary primaries X, Y, and Z
– imaginary = only realizable by spectra that are negative at
some wavelengths
– separates out luminance: X, Z have zero luminance, so Y tells you
the luminance by itself
CIE XYZ Color matching functions
Slide credit: K. Grauman, S. Marschner
Color spaces: CIE XYZ
• Standardized by CIE (Commission Internationale de
l Eclairage, the standards organization for color science)
• Based on three imaginary primaries X, Y, and Z
– imaginary = only realizable by spectra that are negative at
some wavelengths
– separates out luminance: X, Z have zero luminance, so Y tells you
the luminance by itself
Slide credit: K. Grauman, S. Marschner
Perceptually organized color spaces
• Artists often refer to colors as tints, shades, and tones
of pure pigments
– tint: mixture with white
– shade: mixture with black tints
pure
– tones: mixture with white color
black and white
[after Foley et al.]
– gray: no color at all grays
(aka. neutral) shades
black
• This seems intuitive
– tints and shades are inherently related to the pure color
• same color but lighter, darker, paler, etc.
Slide credit: S. Marschner
Perceptual dimensions of color
• Hue
– the kind of color, regardless of attributes
– colorimetric correlate: dominant wavelength
– artist s correlate: the chosen pigment color
• Saturation
– the colorfulness
– colorimetric correlate: purity
– artist s correlate: fraction of paint from the colored tube
• Lightness (or value)
– the overall amount of light
– colorimetric correlate: luminance
– artist s correlate: tints are lighter, shades are darker
Slide credit: S. Marschner
Color spaces: HSV
• Hue, Saturation, Value
• Nonlinear – reflects topology of colors by coding
hue as an angle
Image from mathworks.com Slide credit: K. Grauman
Color spaces: HSV
Intuitive color space
H
(S=1,V=1)
S
(H=1,V=1)
V
(H=1,S=0)
Slide credit: D. Hoiem
Color spaces: YCbCr
Fast to compute, good for
compression, used by TV
Y=0 Y=0.5
Y
(Cb=0.5,Cr=0.5)
Cr
Cb
Cb (Y=0.5,Cr=0.5)
Y=1
Cr
(Y=0.5,Cb=05)
Slide credit: D. Hoiem
Color spaces: YCbCr
Fast to compute, good for
compression, used by TV
Y=0 Y=0.5
Y
(Cb=0.5,Cr=0.5)
Cr
Cb
Cb (Y=0.5,Cr=0.5)
Y=1
Cr
(Y=0.5,Cb=05)
Slide credit: D. Hoiem
Distances in color space
• Are distances between points in a color space
perceptually meaningful?
Slide credit: K. Grauman
Perceptually uniform spaces
• Two major spaces standardized by CIE
– designed so that equal differences in
coordinates produce equally visible
differences in color CIE XYZ
– by remapping color space so that just-
noticeable differences are contained by
circlesà distances more perceptually
meaningful.
– LUV: earlier, simpler space; L*, u*, v*
– LAB: more complex but more uniform:
L*, a*, b*
– both separate luminance from CIE u’v’
chromaticity
– including a gamma-like nonlinear
component is important
Slide credit: K. Grauman, S. Marschner
Color spaces: L*a*b*
“Perceptually uniform”* color space
L
(a=0,b=0)
a
(L=65,b=0)
b
(L=65,a=0)
Slide credit: D. Hoiem
Color spaces: L*a*b*
“Perceptually uniform”* color space
L
(a=0,b=0)
a
(L=65,b=0)
b
(L=65,a=0)
: measured white point
Slide credit: D. Hoiem
Most information in intensity
Only intensity shown – constant color
Slide credit: D. Hoiem
Most information in intensity
Original image
Slide credit: D. Hoiem
Back to grayscale intensity
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
Slide credit: D. Hoiem
Today
• Perception of color and light
• Color spaces
Next week
• Point operations
• Histogram processing
Reading Assignment
• Watch Beau Lotto’s TED talk on “Optical illusions
show how we see”
• Prepare a 1-page summary of the talk
• Due on 17st of October
Programming assignment
• Colorizing the Prokudin-Gorskii photo collection
• A warm-up exercise
• Main steps:
1. Divide the input image into three equal parts
corresponding to RGB channels.
2. Align the second and the third parts (G and R channels)
to the first one (B channel).
Prokudin-Gorskii's Russia in Color
• Russia circa 1900
• One camera, move the film with filters to get 3 exposures
Slide credit: F. Durand
Prokudin-Gorskii's Russia in Color
• Digital restoration
Slide credit: F. Durand
Emir Seyyid Mir Mohammed Alim Khan, the Emir of Bukhara, ca. 1910.
Self-portrait on the Karolitskhali River, ca. 1910.
A metal truss bridge on stone piers, part of the Trans-Siberian Railway,
crossing the Kama River near Perm, Ural Mountains Region, ca. 1910.
On the Sim River, a shepherd boy, ca. 1910.
Peasants harvesting hay in 1909. From the album "Views along the
Mariinskii Canal and river system, Russian Empire", ca. 1910.
