The Glass Class: Designing Wearable Interfaces

The Glass Class
Designing Wearable Interfaces
May 1, CHI 2014
Mark Billinghurst
HIT Lab NZ
University of Canterbury
mark.billinghurst@canterbury.ac.nz
Hayes Raffle
Glass team
Google [x]
hraffle@google.com

Mark Billinghurst
▪ Director of HIT Lab NZ, University
of Canterbury
▪ PhD Univ. Washington
▪ Research on AR, mobile HCI,
Collaborative Interfaces
▪ More than 250 papers in AR, VR,
interface design
▪ Sabbatical in Glass team at
Google [x] in 2013

Hayes Raffle
▪ Interaction Research Lead,
Google Glass
▪ PhD MIT Media Lab
▪ Ran a couple of companies
▪ Launched a few products
▪ Won a few awards
▪ Published many papers in HCI

Course Goals
In this course you will learn
▪ Introduction to head mounted wearable computers
▪ Understanding of current wearable technology
▪ Key design principles/interface metaphors
▪ Relevant human perceptual principles
▪ Rapid prototyping tools
▪ Areas for future research
▪ Hands on experience with the technology

What You Won’t Learn
▪ Low level programming
▪ Glass Mirror API, GDK, Vuzix SDK, etc
▪ Designing for non-HMD based interfaces
▪ Watches, fitness bands, etc
▪ How to develop wearable hardware
▪ optics, sensor assembly, etc
▪ Evaluation methods
▪ Experimental design, statistics, etc

Schedule
9:00 Introduction (Mark + Hayes)
9:05 Overview/History (Mark)
9:20 Evolution and Design Principles (Hayes)
9:45 Prototyping Tools + Best Practices (Mark / Hayes)
10:20 Break/Demo
10:30 Concept Design Exercise (Design group / Hayes)
10:50 Wearable Technologies (Lecture group / Mark)
11:20 Design Presentations (Design group / Hayes)
11:50 Research Directions (Mark + Hayes)
12:20 Finish

Design Group (33 People)
If your name is on this list you are in the Design Group
Yang Wang
Konstantino Kapetaneas
Preethi Srinivas
Tony James
Kate Vogt
Aneesh Tarun
Josh Andres
Maria Maimó
Bram Reurings
Luke Mill
Tuck-Voon How
M Gill
Janaki Kumar
Melinda Knight
M Calkins
Mike Tissenbaum
Samantha Tse
Kal McDowd
Adora Tam
Oscar Meruvia
Mike Chen
Anita Hoechtl
Merlin Stone
Ashoomi Dohlakia
Icy Zhu
Zdenek Mikovec
Cristina Manresa-Yee
Christian Winkler
Angela Noh
Amyris Fernandez
Deborah Ptak
Arne Renkema-Padmos
Thomas Fritz

Display Demos You Can Try
Google Glass Display
Glass UI, AR demos, Games, multimedia capture
Vuzix M-100 Display
Monocular display
Epson BT-100, Epson BT-200
See through displays, Junaio markerless tracking
Brother AirScouter display
Projected see-through image
Recon Snow
Micro-display integrated into ski goggles

CHI Wearables Exhibit
■
Online at http://wcc.gatech.edu/exhibition

A Brief History of Time
▪ Trend
▪ smaller, cheaper, more functions, more intimate
▪ Time pieces moved from public space onto the body
18th
Century
20th
Century
13th
Century

A Brief History of Computing
Trend
▪ Smaller, cheaper, faster, more intimate
▪ Moving from fixed to handheld and onto body
1950’s
1980’s
1990’s

What is a Wearable Computer ?
▪ A computer that is:
▪ Portable while operational
▪ Enables hands-free/hands-limited use
▪ Able to get the user’s attention
▪ Is always on, acting on behalf of the user
▪ Able to sense the user’s current context
Rhodes, B. J. (1997). The wearable remembrance agent: A system for
augmented memory. Personal Technologies, 1(4), 218-224.

In Other Words ..
▪ A computer that is ..
▪ Eudaemonic: User considers it part of him/herself
▪ Existential: User has complete control of the system
▪ Ephemeral: System always operating at some level
Mann, S. (1997). Wearable computing: A first step toward personal
imaging. Computer, 30(2), 25-32.

Wearable Computing
▪ Computer on the body that is:
▪ Always on
▪ Always accessible
▪ Always connected
▪ Other attributes
▪ Augmenting user actions
▪ Aware of user and surroundings

Augmented Interaction
Rekimoto, J., & Nagao, K. (1995, December). The world through the computer:
Computer augmented interaction with real world environments. In Proceedings of the
8th annual ACM symposium on User interface and software technology (pp. 29-36).

The Ideal Wearable
▪ Persists and Provides Constant Access: Designed for
everyday and continuous user over a lifetime.
▪ Senses and Models Context: Observes and models
the users environment, mental state, it’s own state.
▪ Augments and Mediates: Information support for the
user in both the physical and virtual realities.
▪ Interacts Seamlessly: Adapts its input and output
modalities to those most appropriate at the time.
Starner, T. E. (1999). Wearable computing and contextual awareness
(Doctoral dissertation, Massachusetts Institute of Technology).

History of Wearables
▪ 1960-90: Early Exploration
▪ Custom build devices
▪ 1990 - 2000: Academic, Military Research
▪ MIT, CMU, Georgia Tech, EPFL, etc
▪ 1997: ISWC conference starts
▪ 1995 – 2005+: First Commercial Uses
▪ Niche industry applications, Military
▪ 2010 - : Second Wave of Wearables
▪ Consumer applications, Head Worn

Thorp and Shannon (1961)
▪ Wearable timing device for roulette prediction
▪ Audio feedback, four button input
Ed Thorp
Thorp, E. O. (1998, October). The invention of the first wearable computer. In
Wearable Computers, 1998. Second International Symposium on (pp. 4-8). IEEE.

Keith Taft (1972)
▪ Wearable computer for blackjack card counting
▪ Toe input, LED in Glasses for feedback
Belt computer Shoe Input Glasses Display

Steve Mann (1980s - )
http://wearcomp.org/

MIT Wearable Computing (1993-)
http://www.media.mit.edu/wearables/

Enabling Technologies (1989+)
▪ Private Eye Display (Reflection Technologies)
▪ 720 x 280 dipslay
▪ Red LED
▪ Vibrating mirror
▪ Twiddler (Handykey)
▪ Chording keypad
▪ Mouse emulation

MIT Tin Lizzy (1993)
▪ General Purpose Wearable
▪ Doug Platt, Thad Starner
▪ 150 MHz Pentium CPU
▪ 32-64 Mb RAM
▪ 6 Gb hard disk
▪ VGA display
▪ 2 PCMCIA slots
▪ Cellular modem
http://www.media.mit.edu/wearables/lizzy/lizzy/index.html

Early Technology
▪ Computing
▪ Belt or Backpack
▪ Displays
▪ Head Mounted, LCD Panel, Audio
▪ Input Devices
▪ Chording Keyboard, Speech, Camera
▪ Networking
▪ Wireless LAN, Infra-Red, Cellular

US Military Wearables (1989- )
▪ Early experimentation
▪ 386 computer, VGA display
▪ GPS, mapping software
▪ Land Warrior (1991-)
▪ Integrated wearable system
▪ Camera, colour display, radio
▪ Navigation, reports, photos
Zieniewicz, M. J., Johnson, D. C., Wong, C., & Flatt, J. D. (2002). The evolution of
army wearable computers. IEEE Pervasive Computing, 1(4), 30-40.

Wearables at CMU (1991–2000)
▪ Industry focused wearables
▪ Maintenance, repair
▪ Custom designed interface
▪ Dial/button input
▪ Rapid prototyping approach
▪ Industrial designed, ergonomic
http://www.cs.cmu.edu/afs/cs/project/vuman/www/frontpage.html

Early Commercial Systems
▪ Xybernaut (1996 - 2007)
▪ Belt worn, HMD, 200 MHz
▪ ViA (1996 – 2001)
▪ Belt worn, Audio Interface
▪ 700 MHz Crusoe
■ Symbol (1998 – 2006)
■ Wrist worn computer
■ Finger scanner

Prototype Applications
▪ Remembrance Agent
▪ Rhodes (97)
▪ Augmented Reality
▪ Feiner (97), Thomas (98)
▪ Remote Collaboration
▪ Garner (97), Kraut (96)
■ Maintenance
■ Feiner (93), Caudell (92)
▪ Factory Work
▪ Thompson (97)

Mobile AR: Touring Machine (1997)
▪ University of Columbia
▪ Feiner, MacIntyre, Höllerer, Webster
▪ Combines
▪ See through head mounted display
▪ GPS tracking
▪ Orientation sensor
▪ Backpack PC (custom)
▪ Tablet input
Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile
augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.

MARS View
▪ Virtual tags overlaid on the real world
▪ “Information in place”

Backpack/Wearable Systems
1997 Backpack Wearables
▪ Feiner’s Touring Machine
▪ AR Quake (Thomas)
▪ Tinmith (Piekarski)
▪ MCAR (Reitmayr)
▪ Bulky, HMD based
Piekarski, W., & Thomas, B. (2002). ARQuake: the outdoor
augmented reality gaming system. Communications of the
ACM, 45(1), 36-38.

PCI 3D Graphics Board
Hard Drive
Serial
Ports
CPU
PC104 Sound Card
PC104 PCMCIA
GPS
Antenna
RTK correction Antenna
HMD
Controller
Tracker
Controller
DC to DC
Converter
Battery
Wearable
Computer
GPS RTK
correction
Radio
Example self-built working
solution with PCI-based 3D graphics
Columbia Touring Machine
Mobile AR - Hardware

HIT Lab NZ Wearable AR (2004)
▪ Highly accurate outdoor AR
tracking system
▪ GPS, Inertial, RTK system
▪ HMD
▪ First prototype
▪ Laptop based
▪ Video see-through HMD
▪ 2-3 cm tracking accuracy

2008: Location Aware Phones
Nokia NavigatorMotorola Droid

2009 - Layar (www.layar.com)
• Location based data
– GPS + compass location
– Map + camera view
• AR Layers on real world
– Customized data
– Audio, 3D, 2D content
• Easy authoring
• Android, iPhone

Wearable Evolution
Backpack+HMD:
…10+ kg
Handheld + HMD
… Separate sensors
.... UMPC 1.1GHz
…1.5kg
…still >$5K
Scale it down more:
Smartphone…$500
…Integrated
…0.1kg
…billions of units
1997 2003 2007

▪ Hardware
▪ CPU TI OMAP 4430 – 1 Ghz
▪ 16 GB SanDisk Flash,1 GB Ram
▪ 570mAh Battery
▪ Input
▪ 5 mp camera, 720p recording, microphone
▪ GPS, InvenSense MPU-9150 inertial sensor
▪ Output
▪ Bone conducting speaker
▪ 640x360 micro-projector display
Google Glass Specs

Other Wearables
▪ Vuzix M-100
▪ $999, professional
▪ Recon Jet
▪ $600, more sensors, sports
▪ Opinvent
▪ 500 Euro, multi-view mode
▪ Motorola Golden-i
▪ Rugged, remote assistance

Ex: Recon Instruments Snow
Ski display/computer
▪ Location, speed, altitude, phone headset
http://www.reconinstruments.com/

Summary
Wearables are a new class of computing
Intimate, persistent, aware, accessible, connected
Evolution over 50 year history
Backpack to head worn
Custom developed to consumer ready device
Enables new applications
Collaboration, memory, AR, industry, etc
Many head worn wearables are coming
Android based, sensor package, micro-display

Last year Last week NowForever
The Now machine
Focus on location, contextual
and timely information, and
communication.

"Computing
should just be
more comfortable"
"Google should do the hard
work, and you should have a
chance to live, have a good
life, and get on with it."

As technology becomes more
personal and immediate, it can
start to disappear.
Distant Intimate

Early prototyping
Early prototyping

Transparent displays are tricky
Colors are funny and inconsistent.
You can only add light to a scene, not cover anything up.
Motion can be disorienting.
Clarity, contrast, brightness, visual field and attention are important.

Reading
Some things don’t work

Immersion

Details

The world is the experience
Get the interface and interactions out of the way.

Micro
Interactions
The position of the display and
limited input ability makes
longer interactions less
comfortable.
Using it shouldn’t take longer
than taking out your phone.

A rear view mirror
Don't overload the user. Stick to the
absolutely essential, avoid long
interactions. Be explicit.

As personal as
it gets
Recognize and adapt to the
user… not the other way
around.
Glass is the most personal device you own. It
operates closer to your most valuable senses
and your environment, and it knows more about
both.
Glass should adapt, be personal and recognize
the wearer; be aware of what consists his
identity (physiology, memory, preferences,
environment) and connect to it.
As personal as it gets
Glass is the most personal device you own.
It should recognize and adapt to you…
not the other way around.

For the closest people
and most important
moments
Glass should be for prioritizing your
closest people and creating value
for the whole group, not just the
wearer.

Search Picture Messaging PhoneNavigation Video CallVideo

How are people using Glass
for creative expression?

A world of stories
In Viewpoint of Billions by David
Datuna, Glass allows viewers to
unlock images and video with
interactive experiences.

Social action
First-person journalist Tim Pool
broadcasts an intimate view
of Istanbul protests.
'I want to show you what it's like to be there as best I can, even
if that ends with me running full-speed into a cafe and rubbing
lemons all over my face after being tear-gassed'

Sharing and connecting
Conductor Cynthia Johnston Turner
shares a 1st
person experience with her
orchestra.

Personal expression
Alexander Chen’s Viola through Glass.

Our tools are becoming more
intimate and immediate.
We can craft a future of learning, creative
expression and empathy.
Distant Intimate

How can we quickly
prototype Wearable
experiences with little
or no coding?

Why Prototype?
▪ Quick visual design
▪ Capture key interactions
▪ Focus on user experience
▪ Communicate design ideas
▪ “Learn by doing/experiencing”

Prototyping Tools
▪ Static/Low fidelity
▪ Sketching
▪ User interface templates
▪ Storyboards/Application flows
▪ Interactive/High fidelity
▪ Wireframing tools
▪ Mobile prototyping
▪ Native Coding

Important Note
▪ Most current wearables run Android OS
▪ eg Glass, Vuzix, Atheer, Epson, etc
▪ So many tools for prototyping on Android
mobile devices will work for wearables
▪ If you want to learn to code, learn
▪ Java, Android, Javascript/PHP

Typical Development Steps
▪ Sketching
▪ Storyboards
▪ UI Mockups
▪ Interaction Flows
▪ Video Prototypes
▪ Interactive Prototypes
▪ Final Native Application
Increased
Fidelity &
Interactivity

Sketched Interfaces
▪ Sketch + Powerpoint/Photoshop/Illustrator

GlassSim – http://glasssim.com/
▪ Simulate the view through Google Glass
▪ Multiple card templates

GlassSim Card Builder
▪ Use HTML for card details
▪ Multiple templates
▪ Change background
▪ Own image
▪ Camera view

Glass UI Templates
▪ Google Glass Photoshop Templates
▪ http://glass-ui.com/
▪ http://dsky9.com/glassfaq/the-google-glass-psd-template/

Application Storyboard
▪ http://dsky9.com/glassfaq/google-glass-
storyboard-template-download/

ToolKit for Designers
▪ Vectoform Google Glass Toolkit for Designers
▪ http://blog.vectorform.com/2013/09/16/google-glass-
toolkit-for-designers-2/
▪ Sample cards, app flows, icons, etc

Limitations
▪ Positives
▪ Good for documenting screens
▪ Can show application flow
▪ Negatives
▪ No interactivity/transitions
▪ Can’t be used for testing
▪ Can’t deploy on wearable
▪ Can be time consuming to create

▪Series of still photos in a movie format.
▪Demonstrates the experience of the product
▪Discover where concept needs fleshing out.
▪Communicate experience and interface
▪You can use whatever tools, from Flash to iMovie.
Video Sketching

See https://vine.co/v/bgIaLHIpFTB
Example: Video Sketch of Vine UI

Interactive Wireframing
▪ Developing interactive interfaces/wireframes
▪ Transitions, user feedback, interface design
▪ Web based tools
▪ UXpin - http://www.uxpin.com/
▪ proto.io - http://www.proto.io/
▪ Native tools
▪ Justinmind - http://www.justinmind.com/
▪ Axure - http://www.axure.com/

UXpin - www.uxpin.com
▪ Web based wireframing tool
▪ Mobile/Desktop applications
▪ Glass templates, run in browser
https://www.youtube.com/watch?v=0XtS5YP8HcM

Proto.io - http://www.proto.io/
▪ Web based mobile prototyping tool
▪ Features
▪ Prototype for multiple devices
▪ Gesture input, touch events, animations
▪ Share with collaborators
▪ Test on device

Gesture Flow
Scr1
Scr2 Scr3
Scr4 Scr5 Scr6
Ta
p
Swipe

Justinmind - http://www.justinmind.com/
▪ Native wireframing tool
▪ Build mobile apps without programming
▪ drag and drop, interface templates
▪ web based simulation
▪ test on mobile devices
▪ collaborative project sharing
▪ Templates for Glass, custom templates

User Interface - Glass Templates

Comparing Wireframe Tools
Tool Web Native Wearable
Template
Interaction
Uxpin X X
Proto.io X X
Justinmind X X X
Axure X X X

Wireframe Limitations
▪ Can’t deploy on Glass
▪ No access to sensor data
▪ Camera, orientation sensor
▪ No multimedia playback
▪ Audio, video
▪ Simple transitions
▪ No conditional logic
▪ No networking

Processing
▪ Programming tool for Artists/Designers
▪ http://processing.org
▪ Easy to code, Free, Open source, Java based
▪ 2D, 3D, audio/video support
▪ Processing For Android
▪ http://wiki.processing.org/w/Android
▪ Strong Android support
▪ Generates Android .apk file

Processing - Motivation
▪ Language of Interaction
▪ Sketching with code
▪ Support for rich interaction
▪ Large developer community
▪ Active help forums
▪ Dozens of plug-in libraries
▪ Strong Android support
▪ Easy to run on wearables

Basic Parts of a Processing Sketch
/* Notes comment */
//set up global variables
float moveX = 50;
//Initialize the Sketch
void setup (){
}
//draw every frame
void draw(){
}

Importing Libraries
▪ Can add functionality by Importing Libraries
▪ java archives - .jar files
▪ Include import code
import processing.opengl.*;
▪ Popular Libraries
▪ Minim - audio library
▪ OCD - 3D camera views
▪ Physics - physics engine
▪ bluetoothDesktop - bluetooth networking

Processing and Glass
▪ One of the easiest ways to build rich
interactive wearable applications
▪ focus on interactivity, not coding
▪ Collects all sensor input
▪ camera, accelerometer, touch
▪ Can build native Android .apk files
▪ Side load onto Glass

Example: Hello World
//called initially at the start of the Processing sketch
void setup() {
size(640, 360);
background(0);
}
//called every frame to draw output
void draw() {
background(0);
//draw a white text string showing Hello World
fill(255);
text("Hello World", 50, 50);
}

Hello World Image
PImage img; // Create an image variable
void setup() {
size(640, 360);
//load the ok glass home screen image
img = loadImage("okGlass.jpg"); // Load the image into
the program
}
void draw() {
// Displays the image at its actual size at point (0,0)
image(img, 0, 0);
}

Touch Pad Input
▪ Tap recognized as DPAD input
void keyPressed() {
if (key == CODED){
if (keyCode == DPAD) {
// Do something ..
▪ Java code to capture rich motion events
▪ import android.view.MotionEvent;

Motion Event
//Glass Touch Events - reads from touch pad
public boolean dispatchGenericMotionEvent(MotionEvent event) {
float x = event.getX(); // get x/y coords
float y = event.getY();
int action = event.getActionMasked(); // get code for action
switch (action) { // let us know which action code shows up
case MotionEvent.ACTION_DOWN:
touchEvent = "DOWN";
fingerTouch = 1;
break;
case MotionEvent.ACTION_MOVE:
touchEvent = "MOVE";
xpos = myScreenWidth-x*touchPadScaleX;
ypos = y*touchPadScaleY;
break;

Sensors
▪ Ketai Library for Processing
▪ https://code.google.com/p/ketai/
▪ Support all phone sensors
▪ GPS, Compass, Light, Camera, etc
▪ Include Ketai Library
▪ import ketai.sensors.*;
▪ KetaiSensor sensor;

Using Sensors
▪ Setup in Setup( ) function
▪ sensor = new KetaiSensor(this);
▪ sensor.start();
▪ sensor.list();
▪ Event based sensor reading
void onAccelerometerEvent(…)
{
accelerometer.set(x, y, z);
}

Using the Camera
▪ Import camera library
▪ import ketai.camera.*;
▪ KetaiCamera cam;
▪ Setup in Setup( ) function
▪ cam = new KetaiCamera(this, 640, 480, 15);
▪ Draw camera image
void draw() {
//draw the camera image
image(cam, width/2, height/2);
}

Timeline Demo
▪ Create Card Class
▪ load image, card number, children/parent cards
▪ Timeline Demo
▪ Load cards in order
▪ Translate cards with finger motion
▪ Swipe cards in both directions
▪ Snap cards into position

Overview
▪ For best performance need native coding
▪ Low level algorithms etc
▪ Most current wearables based on Android OS
▪ Need Java/Android skills
▪ Many devices have custom API/SDK
▪ Vusix M-100: Vusix SDK
▪ Glass: Mirror API, Glass Developer Kit (GDK)

Glassware and Timeline
▪ Static Cards
▪ Static content with text, HTML, images, and video.
▪ e.g. notification messages, news clip
▪ Live Cards
▪ Dynamic content updated frequently.
▪ e.g. compass, timer
▪ Immersions
▪ Takes over the whole control, out from timeline.
▪ e.g. interactive game

Glassware Development
▪ Mirror API
▪ Server programming, online/web application
▪ Static cards / timeline management
▪ GDK
▪ Android programming, Java (+ C/C++)
▪ Live cards & Immersions
▪ See: https://developers.google.com/glass/

▪ REST API
▪ Java servlet, PHP, Go,
Python, Ruby, .NET
▪ Timeline based apps
▪ Static cards
- Text, HTML, media attachment (image & video)
- Standard and custom menu items
▪ Manage timeline
- Subscribe to timeline notifications
- Sharing with contacts
- Location based services
Mirror API

GDK
▪ Glass Development Kit
▪ Android 4.0.3 ICS + Glass specific APIs
▪ Use standard Android Development Tools

▪ GDK add-on features
▪ Timeline and cards
▪ Menu and UI
▪ Touch pad and gesture
▪ Media (sound, camera and voice input)
GDK

Glass Summary
▪ Use Mirror API if you need ...
▪ Use GDK if you need ...
▪ Or use both

Fake Display
3D print Thingiverse model
see http://www.thingiverse.com/thing:65706
Have the social impact of Google Glass without the cost

Build Your Own Wearable
▪ MyVu display + phone + sensors

Beady-i
▪ http://www.instructables.com/id/DIY-
Google-Glasses-AKA-the-Beady-i/

Rasberry Pi Glasses
▪ Modify video glasses, connect to Rasberry Pi
▪ $200 - $300 in parts, simple assembly
▪ https://learn.adafruit.com/diy-wearable-pi-near-eye-kopin-video-
glasses

Physical Input Devices
▪ Can we develop unobtrusive input devices ?
▪ Reduce need for speech, touch pad input
▪ Socially more acceptable
▪ Examples
▪ Ring,
▪ pendant,
▪ bracelet,
▪ gloves, etc

Prototyping Platform
Arduino Kit Bluetooth Shield Google Glass

Example: Glove Input
▪ Buttons on fingertips
▪ Map touches to commands

Example: Ring Input
▪ Touch strip, button, accelerometer
▪ Tap, swipe, flick actions

How it works
Bracelet
Armband
Gloves
1,2,
3,4
Values/
output

Summary
▪ Prototyping for wearables is similar to mobiles
▪ Tools for UI design, storyboarding, wireframing
▪ Android tools to create interactive prototypes
▪ App Inventor, Processing, etc
▪ Arduino can be used for hardware prototypes
▪ Once prototyped Native Apps can be built
▪ Android + SDK for each platform

Other Tools
▪ Wireframing
▪ pidoco
▪ FluidUI
▪ Rapid Development
▪ Phone Gap
▪ AppMachine
▪ Interactive
▪ App Inventor
▪ WearScript

App Inventor - http://appinventor.mit.edu/
▪ Visual Programming for Android Apps
▪ Features
▪ Access to Android Sensors
▪ Multimedia output
▪ Drag and drop web based interface
▪ Designer view – app layout
▪ Blocks view – program logic/control

Orientation Demo
▪ Use wearable orientation sensor

WearScript
▪ JavaScript development for Glass
▪ http://www.wearscript.com/en/
▪ Script directory
▪ http://weariverse.com/

Best Practices (Dos + don’ts)

✓ Don’t design an app
Glass OS is time-based model, not an app model.
X

✓ Know what makes Glass different than a phone
Glass has certain superpowers. Remember what these
superpowers are and use them to augment the experience you’re
designing.

X
Don’t just port your mobile
experience over to Glass
It won’t work. It will be too busy. It will be hard for users to
quickly understand and navigate your content. The
constraints on Glass are simply too strict.
Bad idea. Even inverted, aside from
the need to properly format each
screen, the layout simply contains
too much information for Glass.
Good idea!

✓ Design for the (hyper)now
When is
my next
meeting?
How many
calories have I
eaten today? Can I
get a burger for
lunch?
Spend 90% of your time thinking about what people want to know
(in sport or elsewhere) at any given moment. The more you know
about what info people need and currently don’t have - the more
compelling your design will be.

✓ Design for emotion
thumbs up viewsnap - running

✓ Make it glanceable
Seek to rigorously reduce information density. Successful designs
afford for recognition, not reading.
Bad Good

✓ Reduce the number of info chunksX✓
You are designing for recognition, not reading. Reducing the total # of
information chunks will greatly increase the glanceability of your design.
1
2
3
1
2
3
4
5 (6)
Test done by Morten Just using a watch

✓ Design single interactions to be faster than 4 s
Eye movements
For 1: 1 230ms
For 2: 1 230ms
For 3: 1 230ms
For 4: 3 (52/17) 690ms
For 5: 2 460ms
~1,840ms
Eye movements
For 1: 1-2 460ms
For 2: 1 230ms
For 3: 1 230ms
~920ms
1
2
3
1
2
3
4
5 (6)
Test done by Morten Just using a watch

Test the glanceability of your design✓

✓ Test your design indoors + outdoors

✓ On the device, black is not blackX
Your mock Device simulation

✓ Establish hierarchy with color - not font size
White is your <h1> and grey is your <h2> or <h3>. Footer text -
establishing time, attribution, or distance - is the only place where
you see a smaller font size used.

✓ If you have brand-specific typography - use it

✓ Use relative information display
calendar card - in 10 minutes, Fri - viewsnap

✓
Remember, people have an ever-growing
ecosystem of wearablesX
Each device should be used when it’s most relevant and when it’s
the easiest interaction available.

✓ The Glass screen is just one part of the
experience

✓ Do view your design on the device
1. Download Android Design Preview
2. Plug in Glass
3. Terminal command: java -jar AndroidDesignPreview-0.3.2.jar
4. Drag the red rectangle over your work to take a look
5. Check out the developer site for more in-depth guidelines

Design Group (33 People)
▪ If your name is on this list you are in the Design Group
Yang Wang
Konstantino Kapetaneas
Preethi Srinivas
Tony James
Kate Vogt
Aneesh Tarun
Josh Andres
Maria Maimó
Bram Reurings
Luke Mill
Tuck-Voon How
M Gill
Janaki Kumar
Melinda Knight
M Calkins
Mike Tissenbaum
Samantha Tse
Kal McDowd
Adora Tam
Oscar Meruvia
Mike Chen
Anita Hoechtl
Merlin Stone
Ashoomi Dohlakia
Icy Zhu
Zdenek Mikovec
Cristina Manresa-Yee
Christian Winkler
Angela Noh
Amyris Fernandez
Deborah Ptak
Arne Renkema-Padmos
Thomas Fritz

Some Key Aspects
▪ Display Technologies
▪ Input Devices
▪ Interaction Metaphors
▪ Perceptual Factors
▪ Ergonomics
▪ Cognitive Aspects

Key Properties of HMD
▪ Field of View
▪ Human eye 95 deg. H, 60/70 deg. V
▪ Resolution
▪ > 320x240 pixel
▪ Refresh Rate
▪ Focus
▪ Fixed/manual
▪ Size, Weight
▪ < 350g for long term
▪ Power

Types of Head Mounted Displays
Occluded
See-thru
Multiplexed

Optical see-through HMD
Virtual images
from monitors
Real
World
Optical
Combiners

Epson Moverio BT-200
▪ Stereo see-through display ($700)
▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g
▪ Android Powered, separate controller
▪ VGA camera, GPS, gyro, accelerometer

Strengths of optical see-through
▪ Simpler (cheaper)
▪ Direct view of real world
▪ Full resolution, no time delay (for real world)
▪ Safety
▪ Lower distortion
▪ No eye displacement
▪ see directly through display

Video see-through HMD
Video
cameras
Monitors
Graphics
Combiner
Video

Vuzix Wrap 1200DXAR
▪ Stereo video see-through display ($1500)
▪ Twin 852 x 480 LCD displays, 35 deg. FOV
▪ Stereo VGA cameras
▪ 3 DOF head tracking

Strengths of Video See-Through
▪ True occlusion
▪ Block image of real world
▪ Digitized image of real world
▪ Flexibility in composition
▪ Matchable time delays
▪ More registration, calibration strategies
▪ Wide FOV is easier to support
▪ wide FOV camera

Multiplexed Displays
▪ Above or below line of sight
▪ Strengths
▪ User has unobstructed view of real world
▪ Simple optics/cheap
▪ Weaknesses
▪ Direct information overlay difficult
▪ Display/camera offset from eyeline
▪ Wide FOV difficult

Vuzix M-100
▪ Monocular multiplexed display ($1000)
▪ 852 x 480 LCD display, 15 deg. FOV
▪ 5 MP camera, HD video
▪ GPS, gyro, accelerometer

Display Types
▪ Curved Mirror
▪ off-axis projection
▪ curved mirrors in front of eye
▪ high distortion, small eye-box
▪ Waveguide
▪ use internal reflection
▪ unobstructed view of world
▪ large eye-box

See-through thin displays
▪ Waveguide techniques for thin see-through displays
▪ Wider FOV, enable AR applications
▪ Social acceptability
Opinvent Ora

Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Holographic
Hologram diffracts light
Limited FOV
Colour bleeding
Diffractive
Slanted gratings
Total internal reflection
Costly, small FOV

Waveguide Methods
See: http://optinvent.com/HUD-HMD-benchmark#benchmarkTable
Clear-Vu Reflective
Several reflective elements
Thinner light guide
Large FOV, eye-box
Reflective
Simple reflective elements
Lower cost
Size is function of FOV

Input Options
▪ Physical Devices
▪ Keyboard
▪ Pointer
▪ Stylus
▪ Natural Input
▪ Speech
▪ Gesture
▪ Other
▪ Physiological

Twiddler Input
▪ Chording or multi-tap input
▪ Possible to achieve 40 - 60 wpm after 30+ hours
▪ Chording input about 50% faster than multi-tap
▪ cf 20 wpm on T9, or 60+ wpm for QWERTY
Lyons, K., Starner, T., Plaisted, D., Fusia, J., Lyons, A., Drew, A., & Looney, E. W. (2004, April).
Twiddler typing: One-handed chording text entry for mobile phones. In Proceedings of the SIGCHI
conference on Human factors in computing systems (pp. 671-678). ACM.

Virtual Keyboards
▪ In air text input
▪ Virtual QWERTY keyboard up to 20 wpm
▪ On real keyboard around 45-60+ wpm
▪ Word Gesture up to 28 wpm
▪ On tablet/phone Word Gesture up to 47 wpm
▪ Handwriting around 20-30 wpm
A. Markussen, et. al. Vulture: A Mid-Air Word-Gesture Keyboard (CHI 2014)

Unobtrusive Input Devices
▪ GestureWrist
▪ Capacitive sensing
▪ Change signal depending on hand shape
Rekimoto, J. (2001). Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In
Wearable Computers, 2001. Proceedings. Fifth International Symposium on (pp. 21-27). IEEE.

Unobtrusive Input Devices
▪ GesturePad
▪ Capacitive multilayered touchpads
▪ Supports interactive clothing

Skinput
Using EMG to detect muscle activity
Tan, D., Morris, D., & Saponas, T. S. (2010). Interfaces on the go. XRDS:
Crossroads, The ACM Magazine for Students, 16(4), 30-34.

Issues to Consider
▪ Fatigue
▪ “Gorrilla” Arm from free-hand input
▪ Comfort
▪ People want to do small gestures by waist
▪ Interaction on the go
▪ Can input be done while moving?

Interaction on the Go
▪ Fitt’s law still applies while interacting on the go
▪ Eg: Tapping while walking reduces speed by > 35%
▪ Increased errors while walking
Lin, M., Goldman, R., Price, K. J., Sears, A., & Jacko, J. (2007). How do people tap when walking? An
empirical investigation of nomadic data entry.International Journal of Human-Computer Studies, 65(9),
759-769.

Information Display
Head Stabilized
Fixed View
Body Stabilized
3 DOF Tracking
World Stabilized
6 DOF Tracking

Spatial Cues for Wearable Info
Billinghurst, M., Bowskill, J., Dyer, N., & Morphett, J. (1998, March). An evaluation of wearable
information spaces. In Virtual Reality Annual International Symposium, 1998. Proceedings., IEEE
1998 (pp. 20-27). IEEE.
▪ Spatial cues sign. improve performance
▪ No difference between audio and visual cues

Spatial Conferencing
3+ attendees can be distinguished with spatialized audio
but could not without spatialized audio
Billinghurst, M., Bowskill, J., Jessop, M., & Morphett, J. (1998, October). A wearable spatial
conferencing space. In Proceedings of ISWC, 1998. (pp. 76-83). IEEE.

Organizing Tools in a Halo Display
Biocca, F., Tang, A., Lamas, D., Gregg, J., Brady, R., & Gai, P. (2001). How do users organize virtual
tools around their body in immersive virtual and augmented environment?: An exploratory study of
egocentric spatial mapping of virtual tools in the mobile infosphere. Media Interface and Network Design
Labs, Michigan State University, East Lansing, MI.

User Attention Metaphors
Cognitive continuums (a) Input, (b) Output
Increase cognitive load from left to right

Notification Interruptions
▪ Gradually increase engagement
▪ Reduce attention load
Receiving SMS on Glass
“Bing”
Tap
Swipe
Glass
Show Message Start Reply
User
Look
Up
Say
Reply

Nomadic Radio (2000)
▪ Spatial audio wearable interface
Sawhney, N., & Schmandt, C. (2000). Nomadic radio: speech and audio interaction for contextual
messaging in nomadic environments. ACM transactions on Computer-Human interaction (TOCHI), 7
(3), 353-383.

Spatial Audio Metaphor
Messages/Events arranged depending on time of day

Notification Interruptions
▪ Dynamic scaling of incoming message
based on interruptibility of the user
▪ Busy = silence
▪ Availble = preview

Layered Audio Notifications
Background ambient audio
Notification scale depending on priority

AR Perceptual Issues
❖ Environment: Issues related to the environment itself.
❖ Capturing: Issues related to digitizing the environment
❖ Augmentation: Issues related to the design, layout, and
registration or AR content
❖ Display device: Technical issues associated with the display
device.
❖ User: Issues associated with user perceiving content.
E. Kruijff, J. E. Swan, and S. Feiner. Perceptual issues in augmented reality revisited.
9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2010, pp. 3--12.

Cognitive Issues in Mobile AR
▪ Three categories of issues
▪ Information Presentation – displaying virtual
information on the real world
▪ Physical Interaction – content creation,
manipulation and navigation
▪ Shared Experience – collaboration and
supporting common experiences
Li, Nai, and Henry Been-Lirn Duh. "Cognitive Issues in Mobile Augmented
Reality: An Embodied Perspective." Human Factors in Augmented Reality
Environments. Springer New York, 2013. 109-135.

Depth Cues
❖ Pictorial: visual cues
• Occlusion, texture, relative brightness
❖ Kinetic: motion cues
• Relative motion parallax, motion perspective
❖ Physiological: motion cues
• Convergence, accommodation
❖ Binocular disparity
• Two different eye images

Use the Following Depth Cues
▪ Movement parallax.
▪ Icon/Object size (for close objects)
▪ Linear perspective
▪ To add side perspective bar.
▪ Overlapping
▪ Works if the objects are big enough
▪ Shades and shadows.
▪ Depends on the available computation

Information Presentation
• Amount of information
• Clutter, complexity
• Representation of information
• Navigation cues, POI representation
• Placement of information
• Head, body, world stabilized
• View combination
• Multiple views

Twitter 360
▪ www.twitter-360.com
▪ iPhone application
▪ See geo-located tweets in real world
▪ Twitter.com supports geo tagging

Wikitude – www.mobilizy.com
Bl
ah
Bl
ah
Bl
ah
Bl
ah Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah
Bl
ah

Information Filtering
•Use context to support information filtering
•Show less information on AR mode (uncluttered screen)

Text Representation
▪ Object space
▪ In general, does not fit wearable applications
▪ Billboards
▪ Screen space
▪ Annotations placed on a 2D plane, usually
parallel to the projection plane.
▪ Better legibility of text.

Outdoor background textures
Changing outdoor illuminanation, and text drawing styles.
Billboard and green text drawing styles are recommended.
Active text drawing styles did not perform better relative to
static styles.

AR Navigation
▪ Problem – how to AR to assist with navigation to POI

Multi-View Navigation
▪ Use multiple views (ego vs exo-centric)
▪ Map/top down view – long distance navigation
▪ AR View – nearby navigation, situational awareness
▪ Radar view in AR view

Transitioning Between Views
▪ Seamless spatial awareness

Resource Competition Framework
▪ Mobility tasks compete for cognitive resources
with other tasks
▪ the most important given higher priority
▪ RCF is a method for analyzing this, based on:
▪ task analysis
▪ modelling cognitive resources
▪ a resource approach to attention
Oulasvirta, A., Tamminen, S., Roto, V., & Kuorelahti, J. (2005, April). Interaction in 4-second bursts: the
fragmented nature of attentional resources in mobile HCI. In Proceedings of the SIGCHI conference on
Human factors in computing systems (pp. 919-928). ACM.

RCF Key Assumptions
Four Key Assumptions
1. Functional Modularity: cognitive system divided into
functionally separate systems with diff. representations
2. Parallel Module Operation: cognitive modules operate
in parallel, independent of each other
3. Limited Capacity: cognitive modules are limited in
capacity with respect to time or content
4. Serial Central Operation: central coordination of
modules (eg monitoring) is serial

Cognitive Interference
▪ Structural interference
▪ Two or more tasks compete for limited
resources of a peripheral system
▪ eg two cognitive processes needing vision
▪ Capacity interference
▪ Total available central processing
overwhelmed by multiple concurrent tasks
▪ eg trying to add and count at same time

Cognitive Resources & Limitations
asdfasdf

Using RCF
1. Map cognitive faculty to task
2. Look for conflicts/overloads
3. Analyse for competition for attention
4. Look for opportunities for technology to
reduce conflicts/competition

Example: Going to work ..
Which is the most cognitively demanding?

Application of RCF
Busy street > Escalator > Café > Laboratory.
But if you made Wayfinding, Path Planning, Estimating Time
to Target, Collision Avoidance easier?

Where to put Wearables?
▪ Places for unobtrusive wearable technology
Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., & Martin, R. (1998, October). Design for wearability. In
Wearable Computers, 1998. Digest of Papers. Second International Symposium on (pp. 116-122). IEEE.

Where to Place Trackpad?
▪ User study 25 people different postures
▪ Front of thigh most preferred, torso/upper arm worst
Thomas, Bruce, et al. "Determination of placement of a body-attached mouse as a pointing input
device for wearable computers." 2012 16th International Symposium on Wearable Computers. IEEE
Computer Society, 1999.

Where do
users want
Wearables?
29% on clothing
28% on wrist
12% on Glasses

Tool/Task Design Paradigm
Lin, R., & Kreifeldt, J. G. (2001). Ergonomics in wearable computer design.
International Journal of Industrial Ergonomics, 27(4), 259-269.

Building on Glass
Design Exercise

Design for Glass
Don’t get in the way
Guidelines

Design for Glass
Keep it timely
Guidelines

Design for Glass
Keep it timely
Avoid the unexpected
Guidelines

Design for Glass
Keep it timely
Avoid the unexpected
Design for people
Guidelines

If you could build anything on Glass,
what would it be?

Sprint - Ideas
Person (mom, family, astronomer)
60 seconds

Sprint - Ideas
Place (in the kitchen, car, hike)
60 seconds

Sprint - Ideas
Function (dance, stargaze, sequence dna)
60 seconds

Sprint - Final deliverable
Make a poster - (Hayes will demo)
Title
One liner
Picture
Pitch
What is it?
What does it do?
Why is it a good idea?

Sprint - Focus
10 min - Ideas to create a product to solve
need

3 min - Share with your group
Sprint - Focus

Sprint - Focus
3 min - Pick a concept

Sprint - Focus
15 min - Dive deeper, develop, poster

Sprint - Focus
15 min - Poster

Sprint - Lightning
2 minute pitch

Challenges for the Future (2001)
▪ Privacy
▪ Power use
▪ Networking
▪ Collaboration
▪ Heat dissipation
▪ Interface design
▪ Intellectual tools
▪ Augmented Reality systems
Starner, T. (2001). The challenges of wearable computing: Part 1. IEEE Micro,21(4), 44-52.
Starner, T. (2001). The challenges of wearable computing: Part 2. IEEE Micro,21(4), 54-67.

Capturing Behaviours
▪ 3 Gear Systems
▪ Kinect/Primesense Sensor
▪ Two hand tracking
▪ http://www.threegear.com

Gesture Interaction With Glass
▪ 3 Gear Systems
▪ Hand tracking
▪ Hand data sent to glass
▪ Wifi networking
▪ Hand joint position
▪ AR application rendering
▪ Vuforia tracking

Performance
▪ Full 3d hand model input
▪ 10 - 15 fps tracking, 1 cm fingertip resolution

User Study
▪ Gesture vs. Touch pad vs. Combined input
▪ Gesture 3x faster, no difference in accuracy

●Meta Gesture Interaction
▪Depth sensor + Stereo see-through
▪https://www.spaceglasses.com/

Ego-Vision Collaboration
▪ Wearable computer
▪ camera + processing + display + connectivity

Current Collaboration
▪ First person remote conferencing/hangouts
▪ Limitations
▪ Single POV, no spatial cues, no annotations, etc

Sharing Space: Social Panoramas
▪ Capture and share social spaces in real time
▪ Enable remote people to feel like they’re with you

Key Technology
▪ Google Glass
▪Capture live panorama (compass + camera)
▪Capture spatial audio, live video
▪ Remote device (desktop, tablet)
▪Immersive viewing, live annotation

Awareness Cues
▪ Where is my partner looking?
▪Enhanced radar display, Context compass

Interaction
▪ Glass Touchpad Input/Tablet Input
▪Shared pointers, Shared drawing

User Evaluation
▪ Key Results
Visual cues significantly increase awareness
Pointing cues preferred for collaboration
Drawing on Glass difficult, ranked low in usability

Context Sensing
▪ Using context to manage information
▪ progressive information display as user shows
interest
▪ Context from
▪ Speech
▪ Gaze
▪ Real world
▪ Wearable AR Display
Ajanki, A., Billinghurst, M., Gamper, H., Järvenpää, T., Kandemir, M., Kaski, S., ... & Tossavainen, T.
(2011). An augmented reality interface to contextual information. Virtual reality, 15(2-3), 161-173.

OpenSource Eyetracker
▪ Could use open source eyetracking
▪ Open Shades Eye Tracker
▪ http://www.wearscript.com/en/latest/eyetracking.html

Glass Resources
▪ Main Developer Website
▪ https://developers.google.com/glass/
▪ Glass Apps Developer Site
▪ http://glass-apps.org/glass-developer
▪ Google Design Guidelines Site
▪ https://developers.google.
com/glass/design/index?utm_source=tuicool
▪ Google Glass Emulator
▪ http://glass-apps.org/google-glass-emulator

Other Resources
▪ AR for Glass Website
▪ http://www.arforglass.org/
▪ Vandrico Database of wearable devices
▪ http://vandrico.com/database

Books
▪ Programming Google Glass
▪ Eric Redmond
▪ Rapid Android Development:
Build Rich, Sensor-Based
Applications with Processing
▪ Daniel Sauter

Contact Details
Mark Billinghurst
▪ email: mark.billinghurst@hitlabnz.org
▪ twitter: @marknb00
Hayes Raffle
▪ email: hraffle@google.com
Feedback + followup form
▪ goo.gl/6SdgzA

The Glass Class: Designing Wearable Interfaces

More Related Content

What's hot

Similar to The Glass Class: Designing Wearable Interfaces

More from Mark Billinghurst

Recently uploaded

The Glass Class: Designing Wearable Interfaces