Lecture 12

Interfaces
COMSATS UNIVERSITY ISLAMABAD WAH CAMPUS
In the previous Lecture
The importance of requirements
Different types of requirements
Data gathering for requirements
Bringing requirements to life
Personas
Scenarios
Capturing interaction with user cases

2
In today’s Lecture
Interface types
Highlight the main design and research considerations for
each of the different interfaces
Consider which interface is best for a given application or
activity

3
20 interface types covered
1. Command 11. Gesture
2. Graphical 12. Haptic
3. Multimedia 13. Multimodal
4. Virtual Reality 14. Shareable
5. Web 15. Tangible
6. Mobile 16. Augumented Reality
7. Appliances 17. Wearbales
8. Voice 18. Robots and Drones
9. Pen 19. Brain-Computer interaction
10. Touch 20. Smart
4
Command line interfaces
Commands such as abbreviations (for instance, ls) typed in at
the prompt to which the system responds (for example, by
listing current files)
Some are hard wired at keyboard, while others can be
assigned to keys
Efficient, precise, and fast
Large overhead to learning set of commands

5
Second Life command line-based interface

6
Research and design considerations

Form, name types and structure are key research questions

Consistency is most important design principle
For example, always use first letter of command
Command interfaces popular for web scripting

7
Graphical user interfaces (GUIs)
Xerox Star first WIMP gave rise to GUIs
Windows
Sections of the screen that can be scrolled, stretched, overlapped, opened, closed,
and moved around the screen using the mouse
Icons
Pictograms that represent applications, objects, commands, and tools that were
opened when clicked on
Menus
Lists of options that can be scrolled through and selected
Pointing device
A mouse controlling the cursor as a point of entry to the windows, menus, and icons
on the screen
8
Example of first generation GUI

9
Simple smartwatch menus with options

10
Windows Design
Windows were invented to overcome the physical constraints
of a computer display
They enable more information to be viewed and tasks to be
performed
Scroll bars within windows enable more information to be
viewed
Multiple windows can make it difficult to find desired one
Listing, tabbing, and thumbnails are techniques that can
help
11
Windows Design: Thumbnails

12
Selecting from a scrolling window

13
Is this method any better?

14
Menu Styles
Flat list: Good for showing large number of options at the same
time when display is small
Drop down: Shows more options on same screen (for example,
cascading)
Pop-up: When pressed, command key for relevant options
Contextual: Provides access to often-used commands
associated with a particular item
Collapsible: Toggles between + and − icons on a header to
expand or contract its contents
Mega: All options shown using 2D drop-down layout
15
Template for a collapsible menu

16
A mega menu

17
Research and design considerations
Window management
Enables users to move fluidly between different windows
(and monitors)
How to switch attention between windows without getting
distracted
Design principles of spacing, grouping, and simplicity should
be used
Which terms to use for menu options (for example, “front”
versus “bring to front”
Mega menus easier to navigate than drop-down ones
18
Icon Design
Icons are assumed to be easier to learn and remember than
commands
Icons can be designed to be compact and variably positioned
on a screen
Now pervasive in every interface
For example, they represent desktop objects, tools (for
example, a paintbrush), applications (for instance, a web
browser), and operations (such as cut, paste, next, accept,
and change)

19
Icon
Since the Xerox Star days, icons have changed in their look
and feel:
black and white
Color, shadowing, photorealistic images, 3D rendering,
and animation
Many designed to be very detailed and animated making them
both visually attractive and informative
Can be highly inviting, emotionally appealing, and feel alive

20
Icon forms
The mapping between the representation and underlying
referent can be:
Similar (for example, a picture of a file to represent the object file)
Analogical (for instance, a picture of a pair of scissors to represent
„cut‟)
Arbitrary (such as the use of an X to represent „delete‟)
The most effective icons are similar ones
Many operations are actions making it more difficult to
represent them
Use a combination of objects and symbols that capture the salient part
of an action
21
2 types of icon styles

22
Flat 2D icons for a smartphone

23
Activity
Sketch simple icons to represent the following operations to
appear on a digital camera screen:
Turn image 90-degrees sideways
Auto-enhance the image
Crop the image
More options
Show them to someone else and see if they can understand
what each represents

24
Basic edit icons on the iPhone app

25
Research and design considerations
There is a wealth of resources for creating icons
Guidelines, style guides, icon builders, libraries, online
tutorials
Text labels can be used alongside icons to help identification
for small icon sets
For large icon sets (for instance, photo editing or word
processing) can use the hover function

26
Multimedia
Combines different media within a single interface with various
forms of interactivity
Graphics, text, video, sound, and animation
Users click on links in an image or text
Another part of the program
An animation or a video clip is played
Users can return to where they were or move on to another place
Can provide better ways of presenting information than a single
media can
27
Pros and Cons
Facilitates rapid access to multiple representations of
information
Can provide better ways of presenting information than can
any media alone
Can enable easier learning, better understanding, more
engagement, and more pleasure
Can encourage users to explore different parts of a game or
story
Tendency to play video clips and animations while skimming
through accompanying text or diagrams
28
Multimedia learning app

29
Research and design considerations
How to design multimedia to help users explore, keep track of,
and integrate the multiple representations
Provide hands-on interactivities and simulations that the
user has to complete to solve a task
Provide quizzes, electronic notebooks, and games
Multimedia good for supporting certain activities, such as
browsing, but less optimal for reading at length

30
Virtual Reality
Computer-generated graphical simulations providing:
“the illusion of participation in a synthetic environment rather
than external observation of such an environment” (Gigante,
1993)
Provide new kinds of experience, enabling users to interact
with objects and navigate in 3D space
Create highly-engaging user experiences

31
Pros and cons
Can have a higher level of fidelity with objects that they
represent compared to multimedia
Induces a sense of presence where someone is totally
engrossed by the experience
“a state of consciousness, the (psychological) sense of
being in the virtual environment” (Slater and Wilbur, 1999)

32
Pros and cons

Provides different viewpoints: first and third person

Early head-mounted displays were uncomfortable to wear and
could cause motion sickness and disorientation
Lighter VR headsets are now available (for example, HTC
Vive) with more accurate head tracking

33
Application Areas
Video games
Arcade games for social groups
Therapy for fears
Experience how others feel emotions
For example, empathy and compassion
Enrich user‟s planning experience for travel destinations
Architecture, design, and education

34
Polygon graphics

35
Research and design considerations

Much research on how to design safe and realistic VRs to

facilitate training
For example, flying simulators
Help people overcome phobias (for example, spiders or talking
in public)

36
Research and design considerations
Design issues
How best to navigate through them (for instance, first versus
third person)
How to control interactions and movements (for example, by
using head and body movements)
How best to interact with information (for instance by using
keypads, pointing, and joystick buttons)
Level of realism to aim for to engender a sense of presence

37
Website design
Early websites were largely text-based, providing hyperlinks
Concern was with how best to structure information to enable
users to navigate and access them easily and quickly
Nowadays, more emphasis is on making pages distinctive,
striking, and aesthetically pleasing
Need to think of how to design information for multiple
platforms—keyboard or touch?
For example, smartphones, tablets, and PCs

38
Usability versus aesthetics?
Vanilla or multi-flavor design?
Ease of finding something versus aesthetic and enjoyable
experience
Web designers are:
“thinking great literature”
Users read the web like a:
“billboard going by at 60 miles an hour” (Krug, 2014)
Need to determine how to brand a web page to catch and keep
„eyeballs‟
39
Breadcrumbs for navigation
Breadcrumbs are category labels:
Enable users to look at other pages without losing track of
where they have come from
Very usable
Enable one-click access to higher site levels
Attract first time visitors to continue to browse a website
having viewed the landing page

40
In your face Web ads

Web advertising is often intrusive and pervasive

Flashing, aggressive, persistent, and annoying
Often requires action to get rid of
What is the alternative?
Use of ad blockers

41
Research and design considerations
Many books and guidelines on website design
Veen‟s (2001) three core questions to consider when designing
any website:
1. Where am I?
2. Where can I go?
3. What‟s here?

42
Activity
Look at a fashion brand’s website, for example, Nike.com
What kind of website is it?
How does it contravene the design principles outlined by Veen?
Does it matter?
What kind of user experience is it providing for?
What was your experience of engaging with it?

43
Mobile interfaces
Handheld devices intended to be used while on the move
Have become pervasive, increasingly used in all aspects of everyday
and working life
For example, phones, fitness trackers, and smartwatches
Larger-sized tablets used in mobile settings
Including those used by flight attendants, marketing professionals,
and at car rental returns

44
QR codes and smartphones

45
Research and design considerations
Mobile interfaces can be cumbersome to use for those with poor
manual dexterity or „fat‟ fingers
Key concern is hit area:
• Area on the phone display that the user touches to make
something happen, such as a key, an icon, a button, or an app
• Space needs to be big enough for all fingers to press accurately
• If too small, the user may accidentally press the wrong key
• Fitts‟ law can be used to help design right spacing
 Minimum tappable areas should be 44 points x 44 points for all controls

46
Appliances
Everyday devices in home, public places, or car
For example, washing machines, remotes, toasters, printers, and navigation
systems)
And personal devices
For instance, digital clock and digital camera
Used for short periods
For example, starting the washing machine, watching a program, buying a
ticket, changing the time, or taking a snapshot
Need to be usable with minimal, if any, learning
47
Simple toaster control

48
Research and design considerations

Need to design as transient interfaces with short interactions

Simple interfaces
Consider trade-off between soft and hard controls
For example, use of buttons or keys, dials, or scrolling

49
Voice User Interfaces
Involves a person talking with a spoken language app, for
example, timetable, travel planner, or phone service
Used most for inquiring about specific information, for example,
flight times or to perform a transaction, such as buying a ticket
Also used by people with visual impairments
For example, speech recognition word processors, page
scanners, web readers, and home control systems

50
Have speech interfaces come of age?

51
Modeling human conversations
People often interrupt each other in a conversation
Especially when ordering in a restaurant, rather than let the
waiter go through all of the options
Speech technology has a similar feature called „barge-in‟
Users can choose an option before the system has finished
listing all of the options available

52
Structuring VUI dialogs
Directed dialogs are where the system is in control of the
conversation
Where it asks specific questions and requires specific responses
More flexible systems allow the user to take the initiative:
For example, “I‟d like to go to Paris next Monday for two weeks.”
But more chance of error, since caller might assume that the system
is like a human
Guided prompts can help callers back on track
For example, “Sorry I did not get all that. Did you say you wanted to fly next
Monday?”
53
Voice assistants (for example, Alexa)
Have become popular in many homes
Allow all to use rather than being single use
Support families playing games, interactive storytelling, jokes,
and so forth
Can encourage social and emotional bonding
Young children (under 4), however, find it difficult to be
understood by the voice assistants
Frustrating for them

54
Research and design considerations
How to design systems that can keep conversation on track
Help people navigate efficiently through a menu system
Enable them to recover easily from errors
Guide those who are vague or ambiguous in their requests for
information or services
Type of voice actor (for example, male, female, neutral, or
dialect)
Do people prefer to listen to and are more patient with a female
or male voice, a northern or southern accent?
55
Pen-based devices
Enable people to write, draw, select, and move objects at an
interface using light pens or styluses
Capitalize on the well-honed drawing skills developed from
childhood
Digital ink, for example, Anoto, use a combination of ordinary
ink pen with digital camera that digitally records everything
written with the pen on special paper

56
The Anoto pen

57
Advantages
Allows users to annotate existing documents quickly and easily
Can be used to fill in paper-based forms that can readily be
converted to a digital record using standard typeface
Can be used by remote teams to communicate and work on
the same documents

58
Touchscreens
Single touchscreens are used in walk-up kiosks (such as ticket
machines and ATMs) to detect the presence and location of a
person‟s touch on the display
Multi-touch surfaces support a range of more dynamic finger tip
actions, for example, swiping, flicking, pinching, pushing, and
tapping

59
Touchscreens (Multi-touch surfaces)
They do so by registering touches at multiple locations using a
grid
Now used for many kinds of displays, such as smartphones,
iPods, tablets, and tabletops
Supports one and two hand gestures, including tapping,
zooming, stretching, flicking, dwelling, and dragging

60
Multi-touch surface

61
Research and design considerations
Provides fluid and direct styles of interaction involving freehand and
pen-based gestures for certain tasks
Core design concerns include whether size, orientation, and shape
of touch displays effect collaboration
Much faster to scroll through wheels, carousels, and bars of
thumbnail images or lists of options by finger flicking
Gestures need to be learned for multi-touch, so a small set of
gestures for common commands is preferable
More cumbersome, error-prone, and slower to type using a virtual
keyboard on a touch display than using a physical keyboard
62
Gesture-based systems
Gestures involve moving arms and hands to communicate
Uses camera recognition, sensor, and computer vision
techniques
Recognize people‟s arm and hand gestures in a room
Gestures need to be presented sequentially to be understood
(compare with the way sentences are constructed)

63
Gestures used in the operating theater

64
Research and design considerations

How does computer recognize and delineate user‟s gestures?

Start and end points?
Difference between deictic and hand waving
How realistic must the mirrored graphical representation of the
user be in order for them to be believable?

65
Haptic interfaces
Provide tactile feedback
By applying vibration and forces to a person‟s body, using
actuators that are embedded in their clothing or a device they
are carrying, such as a smartphone
Vibrotactile feedback can be used to simulate the sense of
touch between remote people who want to communicate
Ultrahaptics creates the illusion of touch in midair using
ultrasound to make the illusion of 3D shapes

66
Realtime vibrotactile feedback
Provides nudges when
playing violin incorrectly
Uses motion capture to
sense arm movements that
deviate from model
Nudges are short vibrations
on arms and hands

67
Exoskeleton with artificial muscles

68
Research and design considerations
Where best to place actuators on body
Whether to use single or sequence of „touches‟
When to buzz and how intense
How does the wearer feel it in different contexts?
What kind of new smartphone/smartwatch apps can use
vibrotactile creatively?
For example, slow tapping to feel like water drops meant to
indicate that it is about to rain, and heavy tapping to indicate a
thunderstorm is looming
69
Multimodal Interfaces
Provide enriched user experiences
By multiplying how information is experienced and detected using different
modalities, such as touch, sight, sound, and speech
Support more flexible, efficient, and expressive means of human-computer
interaction
Most common is speech and vision
Can be combined with multi-sensor input to enable other aspects of
the human body to be tracked
For example, eye gaze, facial expression, and lip movements
Provides input for customizing user interfaces

70
Tracking a person’s movements

Kinect camera can detect multimodal input in real time using RGA camera for facial recognition and gestures,
depth camera for movement tracking, and microphones for voice recognition
Used to build model of person and represented as avatar on display programmed to move just like them

71
Research and design considerations
Need to recognize and analyze user behavior, for example,
speech, gesture, handwriting, or eye gaze
Much harder to calibrate these than single modality systems
What is gained from combining different input and outputs
Is talking and gesturing, as humans do with other humans, a
natural way of interacting with a computer?

72
Shareable interfaces
Designed for more than one person to use:
Provide multiple inputs and sometimes allow simultaneous
input by co-located groups
Large wall displays where people use their own pens or
gestures
Interactive tabletops where small groups interact with
information using their fingertips
For example, DiamondTouch, Smart Table, and Surface

73
A smartboard and an interactive tabletop

74
Benefits
Provide a large interactional space that can support flexible
group working
Can be used by multiple users
Can point to and touch information being displayed
Simultaneously view the interactions and have the same shared
point of reference as others
Can support more equitable participation compared with
groups using single PC

75
Research and design considerations
Core design concerns include whether size, orientation, and
shape of the display have an effect on collaboration
Horizontal surfaces compared with vertical ones support more
turn-taking and collaborative working in co-located groups
Providing larger-sized tabletops does not improve group
working but encourages more division of labor
Having both personal and shared spaces enables groups to
work on their own and in a group
Cross-device systems have been developed to support seamless switching
between these, for example, SurfaceConstellations
76
Tangible Interfaces
Type of sensor-based interaction, where physical objects, for
example, bricks, are coupled with digital representations
When a person manipulates the physical object/s, it causes a
digital effect to occur, for example, an animation
Digital effects can take place in a number of media and places,
or they can be embedded in the physical object

77
Examples
Flow Blocks
Depict changing numbers and lights embedded in the blocks
Vary depending on how they are connected together
Urp
Physical models of buildings moved around on tabletop
Used in combination with tokens for wind and shadows
Digital shadows surrounding them to change over time

78
Examples

MagicCubes
Connect physical electronic components and sensors to make digital
events occur (for example, change color depending on how much
shaken)

79
Learning to code and create with the tangible MagicCubes

80
Benefits
Can be held in one or both hands and combined and
manipulated in ways not possible using other interfaces
Allows for more than one person to explore the interface
together
Objects can be placed on top of each other, beside each
other, and inside each other
Encourages different ways of representing and exploring a
problem space

81
Benefits

People are able to see and understand situations differently

Can lead to greater insight, learning, and problem-solving
than with other kinds of interfaces
Can facilitate creativity and reflection

82
VoxBox
A tangible system that gathers opinions at events through playful and engaging
interaction (Goldsteijn et al., 2015

83
Research and design considerations
What kinds of conceptual frameworks to use to help identify
novel and specific features
What kind of coupling to use between the physical action and
digital effect
If it is to support learning, then an explicit mapping between
action and effect is critical
If it is for entertainment, then it can be better to design it to
be more implicit and unexpected

84
Research and design considerations
What kind of physical artifact to use
Bricks, cubes, and other component sets are most
commonly used because of flexibility and simplicity
Stickies and cardboard tokens can also be used for placing
material onto a surface
With what kinds of digital outputs should tangible interfaces be
combined?

85
Augmented Reality
Augmented reality: Virtual representations are superimposed
on physical devices and objects
Pokémon Go made it a household game
Used smartphone camera and GPS to place virtual
characters onto objects in the environment as if they really
are there
Many other applications including medicine, navigation, air
traffic control, games, and everyday exploring

86
Other Examples
In medicine
Virtual objects, for example, x-rays and scans, are overlaid on part of a
patient‟s body
Aid the physician‟s understanding of what is being examined or
operated
In air traffic control
Dynamic information about aircraft overlaid on a video screen showing
the real planes, and so on landing, taking off, and taxiing
Helps identify planes difficult to make out

87
Augmented reality overlay on a car windshield

88
AR that uses forward facing camera
Enables virtual try-ons (for example, Snapchat filters)
AT mirrors set up in retail stores for trying on make-up,
sunglasses, jewelry
Convenient, engaging, and easy to compare more choices
But cannot feel the weight, texture, or smell of what is being
tried on
Can be used to enable users to step into a character (for
example, David Bowie, Queen Victoria)

89
Other Examples

90
Research and design considerations
What kind of digital augmentation?
When and where in physical environment?
Needs to stand out but not distract from ongoing task
Needs to be able to align with real world objects
What happens if the AR is slightly off?
What kind of device?
Smartphone, tablet, head up display or other?

91
Wearables
First developments were head- and eyewear-mounted
cameras that enabled user to record what was seen and to
access digital information
Since then, jewelry, head-mounted caps, smart fabrics,
glasses, shoes, and jackets have all been used
Provides the user with a means of interacting with digital
information while on the move
Applications include automatic diaries, tour guides, cycle
indicators, and fashion clothing
92
Google Glass (2014)

Why was there so much excitement and concern about people filming what they could see right
in front of them?
93
Research and design considerations
Comfort
Needs to be light, small, not get in the way, fashionable, and
preferably hidden in the clothing
Hygiene
Is it possible to wash or clean the clothing once worn?
Ease of wear
How easy is it to remove the electronic gadgetry and replace it?
Usability
How does the user control the devices that are embedded in the
clothing?
94
Robots
Main types
Remote robots used in hazardous settings
Can be controlled to investigate bombs and other dangerous
materials
Domestic robots helping around the house
Can pick up objects and do daily chores like vacuuming

95
Robots
Main types
Pet robots as human companions
Have therapeutic qualities, helping to reduce stress and loneliness
Sociable robots that work collaboratively with humans
Encourage social behaviors

96
Social robots: Mel and Paro
Cute and cuddly
Can open and close eyes and make sounds and movements

Source: Images courtesy of Mitsubishi Electric Research Labs.

97
Drones
Unmanned aircraft that are controlled remotely and used in a
number of contexts
For example, entertainment, such as carrying drinks and food
to people at festivals and parties
Agricultural applications, such as flying them over vineyards
and fields to collect data about crops, which is useful to
farmers

98
Drones

Helping to track poachers in wildlife parks in Africa

Can fly low and stream photos to a ground station where
images can be stitched together into maps
Can be used to determine the health of a crop, or when it is
the best time to harvest the crop

99
Drone being used to survey

10
0
Research and design considerations
How do humans react to physical robots designed to exhibit
behaviors (for example, making facial expressions) compared
with virtual ones?
Should robots be designed to be human-like or look like and
behave like robots that serve a clearly-defined purpose?
Should the interaction be designed to enable people to
interact with the robot as if it was another human being or
more human-computer-like (for example, pressing buttons to
issue commands)?

10
1
Research and design considerations

Is it acceptable to use unmanned drones to take a series of

images or videos of fields, towns, and private property without
permission or people knowing what is happening?

10
2
Brain-computer interfaces
Brain-computer interfaces (BCI) provide a communication
pathway between a person‟s brain waves and an external
device, such as a cursor on a screen
Person is trained to concentrate on the task, for example,
moving the cursor
BCIs work through detecting changes in the neural functioning
in the brain
BCIs apps:
Games (for example, Brain Ball)
Enable people who are paralyzed to control robots
10
3
BCI being used by a woman

10
4
Smart Interfaces
Smart: phones, speakers, watches, cars, buildings, cites
Smart refers to having some intelligence and connected to the
internet and other devices
Context-aware
Understand what is happening around them and execute appropriate
actions, for example, a Nest thermostat
Human-building interaction
Buildings are designed to sense and act on behalf of the inhabitants
but also allow them to have some control and interaction with the
automated systems
10
5
Which Interfaces
Which interface to use will depend on task, users, context,
cost, robustness, and so on
Is multimedia better than tangible interfaces for learning?
Is speech as effective as a command-based interface?
Is a multimodal interface more effective than a mono-modal
interface?

10
6
Which Interfaces
Will wearable interfaces be better than mobile interfaces for
helping people to find information in foreign cities?
Are virtual environments the ultimate interface for playing
games?
Are shareable interfaces better at supporting communication
and collaboration compared with using networked desktop
PCs?

10
7
Summary
Many innovative interfaces have emerged in last 30 years,
including speech, wearable, mobile, brain, and tangible
This raises many design and research questions as to decide
which to use
For example, how best to represent information to the user so that
they can carry out ongoing activity or task
New smart interfaces that are context-aware and monitor
people
Raising new ethical issues concerned with what data is being
collected and what it is used for
10
8
References

Book: Interaction Design beyond Human Computer Interaction

Chapter 7: Interfaces
7.1 Introduction 7.4 Which Interface?

10
9
In the Next Lectuer

Differences between Goals, and Tasks

Task Analysis
Hierarchical Task Analysis
GOMS

11
0