Niepce, “La Table Servie,” 1822

IMAGE FORMATION

Computer Vision

A whole series of problems

•What is in the image ?
•Object recognition problem
• Where is it ?
•3D spatial layout
• Shape

• How is the camera moving ?

• What is the action ?

Surveillance and security

LOW-LEVEL FEATURE EXTRACTION

Edges, corners

Local regions
 IMAGE IS A PROJECTION OF WORLD

AN UNDER-CONSTRAINED PROBLEM

STEREO VISION

• By having two cameras, we can triangulate features in the left and right images to
obtain depth.

• Need to match features

between the two images:

– Correspondence Problem
 Geometry: 3D models of planar objects

[Fitzgibbon et. al]

[Zisserman et. al. ]

STRUCTURE AND MOTION ESTIMATION

Objective: given a set of images …

Want to compute where the camera is for each

image and the 3D scene structure:

- Uncalibrated cameras

- Automatic estimation from images (no manual clicking)

EXAMPLE

Image sequence Camera path and points

[Fitzgibbon et. al]

[et. al. Zisserman]
APPLICATION: AUGMENTED REALITY

original sequence

AUGMENTATION

INTERPRETATION FROM LIMITED CUES

 SHAPE FROM SHADING
• Recover scene structure from shading in the image

• Typically need to assume:

•Lambertian lighting, isotropic reflectance (cf. infra)

SHAPE FROM TEXTURE

• Texture provides a very strong cue for inferring surface orientation in a single image.

• Necessary to assume homogeneous or isotropic texture.

• Then, it is possible to infer the orientation of surfaces by analyzing how the texture
statistics vary over the image.

SEGMENTATION
Image Segmentation
 HUMAN MOTION DETECTION

Detection: localize the street-lights in the image

Object categorization

mountain

tree

building
banner

street lamp

people vendor
CAMERAS & IMAGE FORMATION

Slides from: F. Durand, S. Seitz, S. Lazebnik, S. Palmer

OVERVIEW
• The pinhole projection model
– Qualitative properties
– Perspective projection matrix
• Cameras with lenses
– Depth of focus
– Field of view
– Lens aberrations

• Digital cameras
– Types of sensors
– Color

LET’S DESIGN A CAMERA

• Idea 1: put a piece of film in front of an object

• Do we get a reasonable image?

Slide by Steve Seitz

 PINHOLE CAMERA

• Add a barrier to block off most of the rays

–This reduces blurring

–The opening is known as the aperture

Slide by Steve Seitz

PINHOLE CAMERA MODEL

• Pinhole model:

– Captures pencil of rays – all rays through a single point

– The point is called Center of Projection (focal point)

– The image is formed on the Image Plane

Slide by Steve Seitz

Slide  source:  Seitz

PROJECTION CAN BE TRICKY…

 Slide  source:  Seitz

PROJECTION CAN BE TRICKY…

CAMERA AND WORLD GEOMETRY

How tall is this woman?

How high is the camera?

What is the camera

rotation?

What is the focal length of

the camera?

Which ball is closer?

PROJECTIVE GEOMETRY
What is lost?

• Length

Who is taller?

Which is closer?
 LENGTH IS NOT PRESERVED

A’
C’

B’

Figure by David Forsyth

PERSPECTIVE EFFECTS

PROJECTIVE GEOMETRY

What is lost?

• Length

• Angles
Parallel?

Perpendicular?
 PROJECTIVE GEOMETRY
What is preserved?
• Straight lines are still straight

VANISHING POINTS

• Each direction in space has its own vanishing point

– All lines going in that direction converge at that point

– Exception: directions parallel to the image plane

• All directions in the same plane have vanishing points

on the same line

PERSPECTIVE DISTORTION

Problem for architectural photography: converging verticals

Source: F. Durand
 PERSPECTIVE AND ART
• Use of correct perspective projection indicated in 1st
century B.C. frescoes

• Skill resurfaces in Renaissance: artists develop systematic

methods to determine perspective projection (around
1480-1515)

Raphael Durer, 1525

K. Grauman

MODELING PROJECTION
y
f

• The coordinate system

– The optical center (O) is at the origin

– The image plane is parallel to xy-plane (perpendicular to z axis)

Source: J. Ponce, S. Seitz

MODELING PROJECTION
y
f

• Projection equations
– Compute intersection with image plane of ray from P = (x,y,z) to O

– Derived using similar triangles

• We get the projection by throwing out the last coordinate:

Source: J. Ponce, S. Seitz

 HOMOGENEOUS COORDINATES

• Is this a linear transformation?

no—division by z is nonlinear

Trick: add one more coordinate:

homogeneous image homogeneous scene

coordinates coordinates

Converting from homogeneous coordinates

Slide by Steve Seitz

PERSPECTIVE PROJECTION MATRIX

• Projection is a matrix multiplication using homogeneous coordinates:

divide by the third

coordinate

PERSPECTIVE PROJECTION & CALIBRATION

• Perspective equations so far in terms of camera’s reference frame….

• Camera’s intrinsic and extrinsic parameters needed to calibrate

geometry.

Camera frame

K. Grauman
 Perspective projection & calibration
World  frame Extrinsic:
Camera  frame  ßàWorld  frame

Intrinsic:
Image  coordinates  rela6ve  to  camera  
ßà Pixel  coordinates
Camera  frame

Camera to World to 3D
2D Perspective
pixel coord. camera coord.
point = projection matrix point
trans. matrix trans. matrix (4x1)
(3x1) (3x4)
(3x3) (4x4)

K.  Grauman

Intrinsic parameters: from idealized world

coordinates to pixel values

Forsyth&Ponce

Perspec6ve  projec6on

W.  Freeman

Intrinsic parameters

But  “pixels”  are  in  some  arbitrary  

spa6al  units

W.  Freeman
 Intrinsic parameters

Maybe  pixels  are  not  square

W.  Freeman

Intrinsic parameters

We  don’t  know  the  origin  of  our  

camera  pixel  coordinates

W.  Freeman

Intrinsic parameters

May  be  skew  between  camera  

pixel  axes

W.  Freeman
 Intrinsic parameters, homogeneous coordinates

Using  homogenous  coordinates,

we  can  write  this  as:

or:

In  pixels
In  camera-­‐based  coords W.  Freeman

Extrinsic parameters: translation

Non-­‐homogeneous  
coordinates

Homogeneous  coordinates

W.  Freeman

Recall, perspective effects…

• Far away objects appear smaller

Forsyth and Ponce

 Perspective effects
• Parallel lines in the scene intersect in the image
• Converge in image on horizon line

Image  plane
(virtual)
pinhole

Scene

PERSPECTIVE EFFECTS

Combining extrinsic and intrinsic calibration

pixels Intrinsic

World  coordinates
Camera  coordinates

Extrinsic

W.  Freeman
 Physical parameters of image
formation
• Geometric
– Type of projection
– Camera pose
• Optical
– Sensor’s lens type
– focal length, field of view, aperture
• Photometric
– Type, direction, intensity of light reaching sensor
– Surfaces’ reflectance properties
• Sensor
– sampling, etc.

Pinhole size / aperture

How does the size of the aperture affect the image we’d
get?

Larger

Smaller

K. Grauman

Adding a lens

focal point

• A lens focuses light onto the film

– Rays passing through the center are not deviated
– All parallel rays converge to one point on a plane located at
the focal length f
Slide by Steve Seitz
 Pinhole vs. lens

K. Grauman

Cameras with lenses

focal point
optical center
(Center Of Projection)

• A lens focuses parallel rays onto a single focal point

• Gather more light, while keeping focus; make pinhole
perspective projection practical

K. Grauman

Thin lens

Thin lens
Rays entering parallel on
one side go through focus
Left focus Right focus on other, and vice versa.
In ideal case – all rays from
P imaged at P’.

Lens diameter d Focal length f

K. Grauman
 Thin lens equation

Any object point satisfying this equation is in focus

K. Grauman

Focus and depth of field

Image  credit:  cambridgeincolour.com

Focus and depth of field

Depth of field: distance between image planes where blur is tolerable

Thin lens: scene points at

distinct depths come in focus
at different image planes.
(Real camera lens systems
have greater depth of field.)

“circles of confusion”

Shapiro and Stockman

 Focus and depth of field
• How does the aperture affect the depth of field?

• A smaller aperture increases the range in which the object is

approximately in focus

Flower images from Wikipedia http://en.wikipedia.org/wiki/Depth_of_field Slide from S. Seitz

Field of view

Angular measure
of portion of 3d
space seen by
the camera

Images from http://en.wikipedia.org/wiki/Angle_of_view

K. Grauman

SAME EFFECT FOR FACES

wide-angle standard telephoto

Source: F. Durand
 Field of view depends on focal length

• As f gets smaller, image

becomes more wide angle
– more world points project onto
the finite image plane
• As f gets larger, image
becomes more telescopic
– smaller part of the world projects
onto the finite image plane

from R. Duraiswami

Field of view depends on focal length

Smaller FOV = larger Focal Length

Slide by A. Efros

Physical parameters of image

formation
• Geometric
– Type of projection
– Camera pose
• Optical
– Sensor’s lens type
– focal length, field of view, aperture
• Photometric
– Type, direction, intensity of light reaching sensor
– Surfaces’ reflectance properties
• Sensor
– sampling, etc.
 Environment map

http://www.sparse.org/3d.html

BDRF

Diffuse / Lambertian
 Foreshortening

Specular reflection

Phong
• Diffuse+specular+ambient:
 Physical parameters of image
formation
• Geometric
– Type of projection
– Camera pose
• Optical
– Sensor’s lens type
– focal length, field of view, aperture
• Photometric
– Type, direction, intensity of light reaching sensor
– Surfaces’ reflectance properties
• Sensor
– sampling, etc.

DIGITAL CAMERA

A digital camera replaces film with a sensor array

– Each cell in the array is light-sensitive diode that converts photons to electrons

– Two common types

• Charge Coupled Device (CCD)

• Complementary metal oxide semiconductor (CMOS)

– http://electronics.howstuffworks.com/digital-camera.htm

Slide by Steve Seitz

Historical context
• Pinhole model: Mozi (470-390 BCE),
Aristotle (384-322 BCE)
• Principles of optics (including lenses):
Alhacen (965-1039 CE) Alhacen’s notes
• Camera obscura: Leonardo da Vinci
(1452-1519), Johann Zahn (1631-1707)
• First photo: Joseph Nicephore Niepce (1822)
• Daguerréotypes (1839)
• Photographic film (Eastman, 1889)
• Cinema (Lumière Brothers, 1895)
Niepce, “La Table Servie,” 1822
• Color Photography (Lumière Brothers, 1908)
• Television (Baird, Farnsworth, Zworykin, 1920s)
• First consumer camera with CCD:
Sony Mavica (1981)
• First fully digital camera: Kodak DCS100 (1990)

Slide credit: L. Lazebnik CCD chip K. Grauman

 Digital images
Think of images as matrices
taken from CCD array.

K. Grauman

Digital images width  520

j=1
i=1
Intensity : [0,255]

500  height

im[176][201]  has  value  164   im[194][203]  has  value  37

K. Grauman
 Color sensing in digital cameras
Bayer grid
Estimate missing components
from neighboring values
(demosaicing)

Why more green?

Human Luminance Sensitivity Function

Source: Steve Seitz

Color images, RGB

color space

R G B

K. Grauman

Physical parameters of image formation

• Geometric
– Type of projection
– Camera pose
• Optical
– Sensor’s lens type
– focal length, field of view, aperture
• Photometric
– Type, direction, intensity of light reaching sensor
– Surfaces’ reflectance properties
• Sensor
– sampling, etc.
 Summary
• Image formation affected by geometry, photometry, and optics.
• Projection equations express how world points map to 2d image.
• Homogenous coordinates allow linear system for projection
equations.
• Lenses make pinhole model practical
• Photometry models: Lambertian, BRDF
• Digital imagers, Bayer demosaicing
Parameters (focal length, aperture, lens diameter, sensor
sampling…) strongly affect image obtained.

K. Grauman