Leow Wee Kheng

CS3249 User Interface Development

Department of Computer Science, SoC, NUS

Images and Video Capture

CS3249 (SoC, NUS) Images and Video Capture 1

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 1
A picture speaks a thousand words...

CS3249 (SoC, NUS) Images and Video Capture 2

CS3249 (SoC, NUS) Images and Video Capture 2
Qt Images
 Qt provides 4 classes for handling images
 QImage
 Optimised for I/O, direct pixel access and manipulation.

 QPixmap
 Optimised for display on screen.

 QBitmap
 Inherited from QPixmap for black/white images.

 QPicture
 Paint device that records and replays QPainter commands.

CS3249 (SoC, NUS) Images and Video Capture 3

CS3249 (SoC, NUS) Images and Video Capture 3
 QLabel provides an easy way to display an image.

#include <QApplication>
#include <QPixmap>
#include <QLabel>

int main(int argv, char **args)

{
QApplication app(argv, args);
QPixmap pic("butterfly.jpg"); // read image
QLabel label;
label.setPixmap(pic); // put image in label
label.show();
return app.exec();
}

CS3249 (SoC, NUS) Images and Video Capture 4

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 4
 QLabel with an image

CS3249 (SoC, NUS) Images and Video Capture 5

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 5
Painting QWidget
 For finer control, directly paint QWidget.
 Create ImageWidget as subclass of QWidget.
 Allow user to zoom image with mouse wheel.
 Need to override two event handlers:
 paintEvent: paints in the widget.

 wheelEvent: processes mouse wheel input.

 Programming ImageWidget is rather tricky.

Need to write specification to make requirements clear.

CS3249 (SoC, NUS) Images and Video Capture 6

CS3249 (SoC, NUS) Images and Video Capture 6
Specification of ImageWidget
 ImageWidget size
 Initial size
 Default depends on platform and screen geometry.

 Set to (800, 600), width = 800, height = 600.

 Minimum size is given by minimumSize().

 Default is (0, 0), which causes widget to disappear.

 Choose (10, 10).

 Client class can change it by calling setMinimumSize().

 Maximum size is given by maximumSize().

 Default is (16777215, 16777215).

 Keep this default.

 Client class can change it by calling setMaximumSize().

CS3249 (SoC, NUS) Images and Video Capture 7

CS3249 (SoC, NUS) Images and Video Capture 7
 Displayed image size
 Displayed image size = zoom  input image size, subject to
minimumSize() ≤ displayed image size ≤ maximumSize()
 ImageWidget size always equals displayed image size.

 setImage(fileName)
 Loads image with file name fileName.
 Initial zoom should be as large as possible but ≤ 1.0
such that displayed image size ≤ initial widget size.
 For image ≤ initial widget size, initial zoom = 1.0.

 For image > initial widget size, initial zoom < 1.0.

CS3249 (SoC, NUS) Images and Video Capture 8

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 8
 Zooming
 Zoom factor > 0.
 Client class can call setZoom(newZoom) to zoom.
 User can zoom in or out using mouse wheel.
 Each mouse wheel step increases or decreases zoom by0.1.

CS3249 (SoC, NUS) Images and Video Capture 9

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 9
// ImageWidget.h

#ifndef IMAGEWIDGET_H
#define IMAGEWIDGET_H

#include <QWidget>
#include <QImage>
#include <QPixmap>

class ImageWidget: public QWidget

{
Q_OBJECT
Q_PROPERTY(float zoom READ getZoom WRITE setZoom)

CS3249 (SoC, NUS) Images and Video Capture 10

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 10
 Property
 Like class variable but has addition features:

Q_PROPERTY(type name
READ getFunction
[WRITE setFunction]
[RESET resetFunction]
[NOTIFY notifySignal]
...
)

 type can be any type supported by QVariant.

 READ declares accessor function.
 WRITE, RESET declare modifier functions (optional).
 NOTIFY declares a signal to be emitted (optional).
 Qt's property system is platform-independent.

CS3249 (SoC, NUS) Images and Video Capture 11

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 11
public:
ImageWidget(QWidget *parent = 0);
void setImage(const QString &fileName);

float getZoom() const { return zoom; } // inline

void setZoom(float newZoom);

protected:
void paintEvent(QPaintEvent *event);
void wheelEvent(QWheelEvent *event);

private:
QPixmap input; // Input image.
QPixmap display; // Displayed image.
float zoom;
};

#endif

CS3249 (SoC, NUS) Images and Video Capture 12

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 12
// ImageWidget.cpp

#include <QtGui>
#include "ImageWidget.h"

ImageWidget::ImageWidget(QWidget *parent): QWidget(parent)

{
// Set defaults
zoom = 0.0;
setMinimumSize(10, 10); // width, height
}

CS3249 (SoC, NUS) Images and Video Capture 13

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 13
void ImageWidget::setImage(const QString &fileName)
{
QImage image = QImage(fileName); // Get image
input = QPixmap::fromImage(image); // For display

// Determine appropriate initial zoom

float initWidth = 800;
float initHeight = 600;
float wRatio = (float) initWidth /
(float) input.width();
float hRatio = (float) initHeight /
(float) input.height();
float initZoom = qMin(wRatio, hRatio);

if (initZoom >= 1.0)

initZoom = 1.0; // Use original size if possible.

setZoom(initZoom);
}

CS3249 (SoC, NUS) Images and Video Capture 14

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 14
void ImageWidget::setZoom(float newZoom)
{
if (newZoom <= 0.0)
return; // no change

QSize newSize = newZoom * input.size();

if (newSize.width() < minimumWidth() ||
newSize.height() < minimumHeight() ||
newSize.width() > maximumWidth() ||
newSize.height() > maximumHeight())
return; // no change

zoom = newZoom;
display = input.scaled(zoom * input.size(),
Qt::KeepAspectRatio);
qDebug("zoom %f", zoom); // for debugging
resize(display.size());
}

CS3249 (SoC, NUS) Images and Video Capture 15

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 15
 input.scaled(...)
 Scales input to zoom * input.size().
 Always scales from input to get optimal image resolution.
Don't rescale the scaled displayed image.
 resize()
 Resizes widget, then sends QResizeEvent to widget.
 Widget's resizeEvent() is called.
 resizeEvent()
 Widget already has new size.
 At the end of the function, sends QPaintEvent to widget.
 Widget's paintEvent() is called.
 Window system may send additional resize events.

CS3249 (SoC, NUS) Images and Video Capture 16

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 16
// Event handlers

void ImageWidget::paintEvent(QPaintEvent *event)

{
QRectF rect = QRectF(QPoint(), size());
QPainter painter(this);
painter.drawPixmap(rect, display, rect);
}

 QRectF(top-left, size) defines a rectangular region.

 QPoint() returns (0, 0).
 painter(this) creates a QPainter for ImageWidget.
 drawPixmap(target, pixmap, source)
draws source part of pixmap into target part of ImageWidget.

CS3249 (SoC, NUS) Images and Video Capture 17

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 17
 Caution
 Calling resize() in resizeEvent() may cause infinite
recursion.
 Calling update() or repaint() in paintEvent() will cause
infinite recursion.

CS3249 (SoC, NUS) Images and Video Capture 18

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 18
void ImageWidget::wheelEvent(QWheelEvent *event)
{
float step = event->delta() / 1200.0;
float newZoom = zoom + step;

char string[100];
sprintf(string, "%.1f", newZoom);
sscanf(string, "%f", &newZoom); // rounding
setZoom(newZoom);
}

 delta() returns integer value in units of 1/8 degree.

 Most mouse wheels rotate in steps of 15 degree,
i.e., 15 / (1/8) = 120 units.
 Divide by 1200.0 to get zoom step of 0.1.

CS3249 (SoC, NUS) Images and Video Capture 19

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 19
// main.cpp

#include <QApplication>
#include "ImageWidget.h"

int main(int argc, char *argv[])

{
QApplication app(argc, argv);
ImageWidget widget;
// widget.setMinimumSize(50, 50);
// widget.setMaximumSize(1100, 700);

if (argc > 1)
widget.setImage(argv[1]);
widget.show();
return app.exec();
}

CS3249 (SoC, NUS) Images and Video Capture 20

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 20
 This example is incomplete:
 Doesn't display an image smaller than minimum widget size.

CS3249 (SoC, NUS) Images and Video Capture 22

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 22
 This example is incomplete:
 Changing widget size has no effect on displayed image.

CS3249 (SoC, NUS) Images and Video Capture 23

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 23
Image is larger than screen?
Two ways to handle large images
 Use scroll area
 Put widget in scroll area.
 Easy to implement.
 Let user pan image
 ImageWidget displays a part of image.
 User uses mouse button to pan image.
 More challenging to implement.
 See UI Challenge in Assignment 2.

CS3249 (SoC, NUS) Images and Video Capture 24

CS3249 (SoC, NUS) Images and Video Capture 24
Using Scroll Area
 QScrollArea
 Subclass of QWidget.
 Provides scrolling view onto another widget.
 Put ImageWidget in QScrollArea
 No change to ImageWidget.
 Scroll bars appear if ImageWidget is too large.

CS3249 (SoC, NUS) Images and Video Capture 25

CS3249 (SoC, NUS) Images and Video Capture 25
// main.cpp
Really easy!
#include <QApplication> There's usually
#include <QScrollArea> an easy way
#include "ImageWidget.h"
to do things in Qt!
int main(int argc, char *argv[])
{
QApplication app(argc, argv);

ImageWidget *widget = new ImageWidget;

if (argc > 1)
widget->setImage(argv[1]);

QScrollArea *scroll = new QScrollArea;

scroll->setWidget(widget);

scroll->show();
return app.exec();
}

CS3249 (SoC, NUS) Images and Video Capture 26

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 26
Interfacing Devices
 Modern computers support many novel devices.

CS3249 (SoC, NUS) Images and Video Capture 29

CS3249 (SoC, NUS) Images and Video Capture 29
 3 basic ways to interface devices:
 Wait for framework developers to develop device interface.
 e.g., QCamera is available in Qt 5.

 How long can you wait?

 Integrate existing tools.

 e.g., OpenCV has camera interface.

 Relatively easy to integrate OpenCV into Qt.

 Develop device drivers.

 Require detailed knowledge.

 Very difficult.

CS3249 (SoC, NUS) Images and Video Capture 30

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 30
Interface Webcam with OpenCV
 Cross-platform open source library for computer vision.
 OpenCV 1 was written in C.
 OpenCV 2 provides C, C++, Python, Java interfaces.
 Supports Linux, Windows, Mac OS, iOS, Andriod.
 Going to support Parallel Javascript.

CS3249 (SoC, NUS) Images and Video Capture 31

CS3249 (SoC, NUS) Images and Video Capture 31
 OpenCV 2 example for video capture with webcam.

// webcam.c

#include <core_c.h>
#include <highgui_c.h>

int main(int argc, char **args)

{
int dev = 0; // 0: default, > 0: other cameras.

cvNamedWindow("webcam", 1); // Create OpenCV window.

CvCapture* capture = cvCaptureFromCAM(dev);

cvSetCaptureProperty(capture,
CV_CAP_PROP_FRAME_WIDTH, 640);
cvSetCaptureProperty(capture,
CV_CAP_PROP_FRAME_HEIGHT, 480);

CS3249 (SoC, NUS) Images and Video Capture 32

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 32
 while(1)
{
IplImage *frame = cvQueryFrame(capture);

cvShowImage("webcam", frame);

char key = cvWaitKey(10);

if (key == 27) // If key is ESC,
break; // break out of loop.
}

cvReleaseCapture(&capture);
cvDestroyWindow("webcam");
return 0;
}

CS3249 (SoC, NUS) Images and Video Capture 33

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 33
Integrating OpenCV in Qt
 OpenCV video capture program is sequential.
 Qt is event-driven.
 To integrate OpenCV in Qt:
 Use Qt timer event to generate regularly timed events.
 Call OpenCV function upon receiving timer event.
 Convert OpenCV image to QImage for display.

CS3249 (SoC, NUS) Images and Video Capture 35

CS3249 (SoC, NUS) Images and Video Capture 35
 Integrate OpenCV video capture functions in Qt.

Native Webcam
screen display (QWidget)
interface output

QTimer
init timer
OpenCV
video capture OpenCV event
get frame
functions interface
frame

CS3249 (SoC, NUS) Images and Video Capture 36

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 36
// Webcam.h

#ifndef WEBCAM_H
#define WEBCAM_H

#include <QWidget>
#include <QTime>
#include <core_c.h>
#include <highgui_c.h>

class Webcam: public QWidget

{
Q_OBJECT

public:
Webcam(int dev, int fps);
~Webcam();

CS3249 (SoC, NUS) Images and Video Capture 37

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 37
protected:
void showEvent(QShowEvent *event);
void hideEvent(QHideEvent *event);
void timerEvent(QTimerEvent *event);
void paintEvent(QPaintEvent *event);

private:
CvCapture* capture;
int timerId;
int frameRate; // input frame rate
QPixmap pixmap;
int nframes; // used to calculate actual frame rate
QTime time; // used to calculate actual frame rate

// convert image format

QImage IplImage2QImage(const IplImage *iplImage);
};

#endif

CS3249 (SoC, NUS) Images and Video Capture 38

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 38
// Webcam.cpp

#include <QtGui>
#include "Webcam.h"

Webcam::Webcam(int dev, int fps)

{
timerId = 0;
frameRate = fps;
capture = cvCaptureFromCAM(dev);

int width = 640;

int height = 480;
cvSetCaptureProperty(capture,
CV_CAP_PROP_FRAME_WIDTH, width);
cvSetCaptureProperty(capture,
CV_CAP_PROP_FRAME_HEIGHT, height);
setFixedSize(width, height);
}

CS3249 (SoC, NUS) Images and Video Capture 39

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 39
Webcam::~Webcam()
{
cvReleaseCapture(&capture);
}

void Webcam::showEvent(QShowEvent *event)

{
nframes = 0; // init
timerId = startTimer(1000 / frameRate); // in msec
time.start(); // start time
}

void Webcam::hideEvent(QHideEvent *event)

{
killTimer(timerId);
}

CS3249 (SoC, NUS) Images and Video Capture 40

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 40
void Webcam::timerEvent(QTimerEvent *event)
{
if (event->timerId() == timerId)
{
IplImage *frame = cvQueryFrame(capture);
QImage image = IplImage2QImage(frame); // convert
pixmap = QPixmap::fromImage(image); // convert
repaint(); // immediate repaint

if (++nframes == 50)
{
qDebug("frame rate: %f", // actual frame rate
(float) nframes * 1000 / time.elapsed());
nframes = 0;
time.restart();
}
}
else
QWidget::timerEvent(event);
}

CS3249 (SoC, NUS) Images and Video Capture 41

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 41
void Webcam::paintEvent(QPaintEvent *event)
{
QRectF rect = QRectF(QPoint(), size());
QPainter painter(this);
painter.drawPixmap(rect, pixmap, rect);
}

CS3249 (SoC, NUS) Images and Video Capture 42

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 42
// Convert OpenCV's IplImage to QImage.

QImage Webcam::IplImage2QImage(const IplImage *iplImage)

{
int height = iplImage->height;
int width = iplImage->width;

if(iplImage->depth == IPL_DEPTH_8U &&

iplImage->nChannels == 3) // colour image
{
const uchar *qImageBuffer =
(const uchar*) iplImage->imageData;
QImage img(qImageBuffer, width, height,
QImage::Format_RGB888);
return img.rgbSwapped();
}

CS3249 (SoC, NUS) Images and Video Capture 43

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 43
 else if(iplImage->depth == IPL_DEPTH_8U &&
iplImage->nChannels == 1) // gray image
{
const uchar *qImageBuffer =
(const uchar*) iplImage->imageData;
QImage img(qImageBuffer, width, height,
QImage::Format_Indexed8);

QVector<QRgb> colorTable; // set up colour table

for (int i = 0; i < 256; i++)
colorTable.append(qRgb(i, i, i));

img.setColorTable(colorTable);
return img;
}
else
{
qWarning() << "Image cannot be converted.";
return QImage();
}
}
CS3249 (SoC, NUS) Images and Video Capture 44
CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 44
// main.cpp

#include <QApplication>
#include "Webcam.h"
#include "stdio.h"

int main(int argc, char **args)

{
QApplication app(argc, args);

int dev = 0; // default device

int fps = 25; // 25 frames per second
if (argc > 1)
sscanf(args[1], "%d", &dev);
if (argc > 2)
sscanf(args[2], "%d", &fps);

Webcam *view = new Webcam(dev, fps);

view->show();
return app.exec();
}
CS3249 (SoC, NUS) Images and Video Capture 45
CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 45
# webcam.pro
# Run qmake -project to generate file.
# Then, add the include line.
# Then, run qmake followed by make.

TEMPLATE = app
TARGET =
DEPENDPATH += .
INCLUDEPATH += .

# include additional compiler info for OpenCV

include(opencv.pro)

# Input
HEADERS += Webcam.h
SOURCES += main.cpp Webcam.cpp

CS3249 (SoC, NUS) Images and Video Capture 46

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 46
# opencv.pro

QMAKE_CXXFLAGS += -Wno-deprecated
INCLUDEPATH += /usr/include/opencv2/core
/usr/include/opencv2/highgui
LIBS += -L/usr/lib -lopencv_core -lopencv_imgproc
-lopencv_highgui

CS3249 (SoC, NUS) Images and Video Capture 47

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 47
 Notes on video capture
 Most cameras have a max frame rate, e.g., 30 fps.
 When desired frame rate > max frame rate,
program can only run at max frame rate.

CS3249 (SoC, NUS) Images and Video Capture 48

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 48
Real-Time Video Processing

Image
Processor

Native Webcam
screen display (QWidget)
interface output processed
image

QTimer
init timer
OpenCV
video capture OpenCV event
get frame
functions interface
frame
CS3249 (SoC, NUS) Images and Video Capture 49
CS3249 (SoC, NUS) Images and Video Capture 49
// ImageProc.h

#ifndef IMAGEPROC_H
#define IMAGEPROC_H

#include <core_c.h>
#include <highgui_c.h>

class ImageProc
{
public:
IplImage *process(IplImage *source, int type);
};

#endif

CS3249 (SoC, NUS) Images and Video Capture 50

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 50
// ImageProc.cpp

#include "ImageProc.h"
#include <imgproc_c.h>

IplImage *ImageProc::process(IplImage *source, int type)

{
static IplImage *target = NULL;

if (target)
cvReleaseImage(&target);

target = cvCreateImage(
cvSize(source->width, source->height),
source->depth, source->nChannels);

CS3249 (SoC, NUS) Images and Video Capture 51

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 51
 switch(type)
{
case 0: // do nothing
return source;

case 1: // demo fast algo: fast smoothing

cvSmooth(source, target, CV_GAUSSIAN, 5);
return target;

CS3249 (SoC, NUS) Images and Video Capture 52

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 52
 case 2: // demo slow algo: filter with large kernel
{
int size = 100;
CvMat *kernel = cvCreateMat(size, size,
CV_32FC1);
cvSet(kernel, cvScalar(1.0 / size / size));
cvFilter2D(source, target, kernel);
cvReleaseMat(&kernel);

// Filter target is too blurred for display.

// Display the following for demo purpose.
cvSmooth(source, target, CV_GAUSSIAN, 5);
return target;
}

default: // do nothing
return source;
}
}

CS3249 (SoC, NUS) Images and Video Capture 53

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 53
// Webcam.h

class Webcam: public QWidget

{
Q_OBJECT

public:
Webcam(int dev, int fps, int type);
~Webcam();
...

protected:
...
void timerEvent(QTimerEvent *event);

private:
...
ImageProc *imageProc;
int demoType;
};

CS3249 (SoC, NUS) Images and Video Capture 54

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 54
// Webcam.cpp

Webcam::Webcam(int dev, int fps, int type)

{
timerId = 0;
frameRate = fps;
demoType = type;

capture = cvCaptureFromCAM(dev);

int width = 640;

int height = 480;
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH,
width);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT,
height);
setFixedSize(width, height);

imageProc = new ImageProc;

}

CS3249 (SoC, NUS) Images and Video Capture 55

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 55
// Webcam.cpp

Webcam::~Webcam()
{
cvReleaseCapture(&capture);
delete imageProc;
}

CS3249 (SoC, NUS) Images and Video Capture 56

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 56
void Webcam::timerEvent(QTimerEvent *event)
{
if (event->timerId() == timerId)
{
IplImage *frame = cvQueryFrame(capture);
IplImage *target =
imageProc->process(frame, demoType);
QImage image = IplImage2QImage(target);
pixmap = QPixmap::fromImage(image);
repaint();

...
}
}

CS3249 (SoC, NUS) Images and Video Capture 57

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 57
Response Time
 Let desired frame rate = 25 fps.
 Frame interval τ = 1 / 25 sec = 40 msec.
 Timer event
 ck : time at which timer event arrives.
 Video capture
 fk : time at which video frame k is captured.
 Frame processing
 dk : time at which frame k is displayed after processing.

CS3249 (SoC, NUS) Images and Video Capture 58

CS3249 (SoC, NUS) Images and Video Capture 58
 Response time
 Frame capture time = fk – ck.
 Frame processing time = dk – fk .
 Response time rk = dk – ck.
 Want rk ≤ τ.

CS3249 (SoC, NUS) Images and Video Capture 59

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 59
 Fast processing
 Response time r < τ
 Actual frame rate = desired frame rate.

CS3249 (SoC, NUS) Images and Video Capture 60

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 60
 Slower processing
 Response time r ≈ 2τ.
 Actual frame rate ≈ 1/2 desired frame rate.
 Slower rate but still responsive.

 Question:
 Timer events can fire at desired frame rate.
What happen to missing timer events (dotted arrows)?

CS3249 (SoC, NUS) Images and Video Capture 61

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 61
General Software Architecture
 Better to split input and output modules.

output

Native Webcam
screen display (QWidget)
interface output

QTimer
init timer
OpenCV
video capture OpenCV event
get frame
functions interface
frame
input
CS3249 (SoC, NUS) Images and Video Capture 63
CS3249 (SoC, NUS) Images and Video Capture 63
 processed
image
Native Main
display
screen Window
output
interface
QTimer Image
Processor
generic
init camera timer
functions event
init get frame
frame

init
OpenCV Webcam
video capture get frame (camera
functions interface)
frame

CS3249 (SoC, NUS) Images and Video Capture 64

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 64
 processed
image
Main
display
Window
output
QTimer Image
Processor
generic
init camera timer
simplified functions event
diagram
init get frame
frame

init
OpenCV Webcam
video capture get frame (camera
functions interface)
frame

CS3249 (SoC, NUS) Images and Video Capture 65

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 65
 processed
image
Main
display
Window
output
Image
Processor
generic
camera
Which pattern functions
is this?
init get frame
frame

init
OpenCV Webcam
video capture get frame (camera
functions interface)
frame

CS3249 (SoC, NUS) Images and Video Capture 66

CS3249
CS3249 (SoC, NUS) (SoC, NUS) Images and Video Capture 66
Summary
 Working with images
 Use QImage for input/output.
 Use QPixmap for painting on QWidget.
 Reimplement event handlers
 paintEvent(), wheelEvent(), etc.

 Write specification to clarity requirements.

 Integrating devices
 Use existing tools with hardware interfaces.
 Integrate tools into Qt.
 Need to convert data to/from Qt format.
 Check response time.

CS3249 (SoC, NUS) Images and Video Capture 67

CS3249 (SoC, NUS) Images and Video Capture 67
Further Reading
 2D and 3D graphics: [Blan2008], Chap. 8.

CS3249 (SoC, NUS) Images and Video Capture 68

CS3249 (SoC, NUS) Images and Video Capture 68
Exercise
 Modify the sample codes into 3 loosely coupled parts:
 Main window: displays video frame and interacts with user.
 Webcam: captures video frame.
 ImageProc: processes images.

CS3249 (SoC, NUS) Images and Video Capture 69

CS3249 (SoC, NUS) Images and Video Capture 69
References
 J. Blanchette and M. Summerfield, C++ GUI Programming with
Qt 4, 2nd ed., Prentice Hall, 2008.
 OpenCV Reference Manual.

CS3249 (SoC, NUS) Images and Video Capture 70

CS3249 (SoC, NUS) Images and Video Capture 70