DATA REPRESENTATION (PART 3)
MULTIMEDIA
MULTIMEDIA DATA
• Multimedia means different types of data
• Text
• Images
• Audio
• Video
• Graphics
• Already seen how text is represented (ASCII)
• Image vs. vector graphics
CONTINUOUS VS. DISCRETE
• Most ‘real life’ data is continuous
• Takes on any value in a range of values (i.e. infinite possibilities)
• How loud is a piece of music?
• How bright is a light?
• Computers represent data as discrete values
• Values are separate and distinct with gaps between them
• 1, 2, 3, 4, 5, 6, …
DIGITALIZATION, QUANTIZATION, AND SAMPLING
• Digitization – converting from continuous data to discrete data
• Quantization – Digitization of amplitude – the size (loudness, brightness, etc.)
• Sampling – How often measurements are made (sounds) or how close the
measurements are (images)
AUDIO DATA
• Sounds are produced by vibrations
• Frequency – how often vibration
• Amplitude – Size of vibration
• Sounds are sampled at discrete times and amplitude is measured
• Amplitude is quantized, usually an integer
• CD quality
• 44100 samples/second (44Hz)
• Sample represented as 16 bits (65536 values)
• 1 second audio = 44100 16-bit numbers (88 kB)
IMAGE DATA
• A digital camera samples an image with light sensors.
• Each sample is a pixel.
• Pixels are arranged in a rectangular array, M rows and N columns.
• The measurement at each pixel is the brightness for black and white images.
• Measurements are usually an integer in the range 0–255 (8 bit). 0 means
black and 255 means white.
IMAGE DATA
• For colour images, three measurements are made:
• How much red, green, and blue light is in that position (RGB).
• An image is represented in a computer as an array of pixels (usually one row
at a time). Each pixel is represented by either one number (black and white)
or three numbers (colour).
• Colours are commonly defined by the RGB values in hexadecimal (e.g. HTML).
e.g. Yellow is #FFFF00
• Equal amounts of R, G, B gives black (#000000), white (#FFFFFF), and various
shades of gray.
VIDEO DATA
• Videos are essentially a sequence of images sampled close in time.
• Each image is called a frame.
• Traditional film movies have 24 frames each second.
• HDTV at 1080p may have 24, 50, 60 frames each second.
• A video is represented in a computer as a sequence of images.
SAMPLING RATE
• If we sample too often, there is too much processing and more data is
produced.
• If we don’t sample enough, the audio/video may be “choppy.”
• We may even be tricked into seeing something else (aliasing).
ALIASING
Original Scaled down by a factor of 2
Notice the direction of the stripes when the image is scaled down.
UNDERSAMPLED: ALIASING
• Top-secret helicopter?
• Car chases in movies
• Explanation
SAMPLING
• Nyquist theorem: we must sample at least twice as fast as the maximum
frequency to obtain an accurate representation.
• Intuitively, how often you sample depends on how quickly the data change.
• Sampling at least twice as fast as the maximum frequency ensures that a
change is not missed.
• Conversely a signal that does not change (constant) only has to be sampled
once.
• Human can hear sounds up to about 20000Hz, hence the sampling rate of CD
audio is about twice as much.
HOW MUCH DATA?
• 2 minutes of CD quality audio: 10.6 MB
• iPhone 6 picture (8 megapixels): 24 MB
• 1 hour of HDTV 1080p video (24fps): 501 GB
• That’s a lot of data!
HOW MUCH DATA?
• Capacity of CD: about 700MB—a little over 2 hours
• Capacity of DVD: 4.7GB—33 seconds of 1080p video, with no audio!
• DVD movies are generally lower quality than 1080p. Luckily the data can be
compressed.
DATA VS. INFORMATION
• Although there is a lot of data, there may not be much information.
• Intuitively, a piece of data has a lot of information if we learn something
unexpected from the data.
• “Everyone who writes the final exam will get at least 0%.” is not very informative—you
already know this. The data is redundant.
• “It snows in August.” is very informative because it is not expected.
• Compression works by removing redundancy, in order to obtain a more
compact representation.
COMPRESSION
• Compression works by removing certain redundancies. A certain context is needed
(e.g. what you already know).
• Images: the colour of a pixel is likely similar to those close by. If you know one pixel it is likely
you know what is close by.
• Videos: the colour of a pixel is likely similar to those in a previous frame.
• Many compression algorithms work by “prediction”: only record prediction errors.
• Run length encoding is a method to compress data with many consecutive repeated
values.
LOSSY COMPRESSION
• Most practical compression algorithms are lossy - the data stored is a close
approximation of the original.
• By accepting possible loss, we may be able to compress more.
• Instead of using 8 bits to represent 256 possible levels, use 7 bits to represent 128 levels.
• We save 12.5% space but the data is not as accurate.
• Why does this work? We may not be able to tell the difference!
• There is a trade-off between quality and space usage. That is often controlled by a
“quality” setting in your digital camera, for example.
JPEG COMPRESSION
• Many images are compressed using the JPEG standard (.jpg).
• Developed by the Joint Photographic Experts Group in 1992.
• Designed for photographs, lossy.
• A quality setting is used to control space and quality.
• It uses a variety of techniques, including run-length encoding.
• There are many newer algorithms but JPEG is still one of the most widely
used.
DIFFERENT JPG QUALITIES
8 KB 14 KB 21 KB 144 KB
BASIC IMAGE PROCESSING
• Colour balance: the three colour components (R,G,B) can be manipulated
independently to change the colour balance.
• If there is too much yellow, we can decrease the R and G values, or we can also increase the B
values.
• What is the difference between the two approaches above?
• Contrast enhancement
• Blur out certain parts—tend to remove details.
• Sharpening certain parts—tend to highlight details
CONTRAST ENHANCEMENT
• Contrast: relative difference between two tones
BLUR / SHARPEN
• Blur out certain parts—tend to remove details.
• Sharpening certain parts—tend to highlight details
MPG3
• Lossy compression of audio files
• Standard for portable digital players
• An MP3 file created using 128 kbits/s results in a file 1/11 the size of the CD source
• Uses perceptual coding. Removes:
• Sounds (most) humans can’t hear
• Quieter of two sounds that happen at the same time
VIDEO CODECS AND CONTAINERS
• Video compression predicts differences between frames
• Many different video codecs (encoder/decoder)
• MPEG-4 and H.264 most popular video codecs
• Containers
• Combination of video codec and audio codec in one package
• Also contains other information (e.g. titles/chapters, subtitles, other languages for
DVD/Blu-ray)
• For many applications (e.g. Facebook photos), lossy compression is
acceptable.
• Are there applications in which lossy compression is not acceptable?
• What are the implications of using lossy compression?
