426 Lecture 8: Mobile Augmented Reality

COSC 426: Augmented Reality

Mark Billinghurst
mark.billinghurst@hitlabnz.org

Sept 12th 2012

Lecture 8: Mobile Augmented Reality

1998: SGI O2 2008: Nokia N95

CPU: 300 Mhz CPU: 332 Mhz
HDD; 9GB HDD; 8GB
RAM: 512 mb RAM: 128 mb
Camera: VGA 30fps Camera: VGA 30 fps
Graphics: 500K poly/sec Graphics: 2m poly/sec

Mobile Phone AR
  Mobile Phones
  camera
  processor
  display
  AR on Mobile Phones
  Simple graphics
  Optimized computer vision
  Collaborative Interaction

2005: Collaborative AR

  AR Tennis
  Shared AR content
  Two user game
  Audio + haptic feedback
  Bluetooth networking

Evolution of Mobile AR
Camera phone
Camera phone
- Thin client AR
Wearable Wearable AR
Computers Camera phone
- Self contained AR
PDAs
Handheld -Thin client AR
AR Displays
PDAs
-Self contained AR

1995 1997 2001 2003 2004

Handheld Displays
Tethered Applications
  Fitzmaurice Chameleon (1994)
  Rekimoto’s Transvision (1995)
  Tethered LCD
  PC Processing and Tracking

Handheld AR Display - Tethered
1995, 1996 Handheld AR
  ARPad, Cameleon
  Rekimoto’s NaviCam, Transvision
  Tethered LCD
  PC Processing and Tracking

NaviCam (Rekimoto, 1995)
Information is registered to
real-world context
  Hand held AR displays
Interaction
  Manipulation of a window
into information space
Applications
  Context-aware information displays

AR Pad (Mogilev 2002)
Handheld AR Display
  LCD screen
  Camera
  SpaceOrb 3 DOF controller
  Peripheral awareness
  Viewpoint awareness

Mobile AR: Touring Machine (1997)
  University of Columbia
  Feiner, MacIntyre, Höllerer, Webster
  Combines
  See through head mounted display
  GPS tracking
  Orientation sensor
  Backpack PC (custom)
  Tablet input

MARS View

  Virtual tags overlaid on the real world
  “Information in place”

Backpack/Wearable AR
1997 Backpack AR
  Feiner’s Touring Machine
  AR Quake (Thomas)
  Tinmith (Piekarski)
  MCAR (Reitmayr)
  Bulky, HMD based

Mobile AR - Hardware
GPS
Example self-built working
Antenna
solution with PCI-based 3D graphics

PCI 3D Graphics Board

Tracker
Controller
PC104 Sound Card

DC to DC
Wearable
Converter CPU
Computer
PC104 PCMCIA

Battery

GPS RTK Hard Drive
correction
Radio

Serial
Ports

Columbia Touring Machine

Sharp J-SH04
  1997 Philip Kahn invents camera phone
  1999 First commercial camera phone

Handheld AR – Thin Client
2001 BatPortal (AT&T Cambridge)
  PDA used as I/O device
  Wireless connection to workstation
  Room-scale ultrasonic tracking (Bat)

2001 AR-PDA (C Lab)
  PDA thin graphics client
  Remote image processing
  www.ar-pda.com

Mobile Phone AR – Thin Client

2003 ARphone (Univ. of Sydney)
  Transfer images via Bluetooth (slow – 30 sec/image)
  Remote processing – AR Server
 


Early Phone Computer Vision Apps
2003 – Mozzies Game - Best mobile game
Optical motion flow detecting phone orientation
Siemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH)
Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)

Computer Vision on Mobile Phone
  Cameras and Phone CPU sufficient for computer vision
applications
  Pattern Recognition (Static Processing)
  QR Code
  Shotcode (http://www.shotcode.com/)
  Motion Flow (2D Image Processing)
  GestureTek
-  http://www.gesturetekmobile.com/
  TinyMotion
  3D Pose Calculation
  Augmented Reality

Handheld AR – Self Contained
2003 PDA-based AR
  ARToolKit port to PDA
  Studierstube ported to PDA
  AR Kanji Educational App.
  Mr Virtuoso AR character
  Wagner’s Invisible Train
-  Collaborative AR

Mobile Phone AR – Self Contained
2004 Mobile Phone AR
  Moehring, Bimber
  Henrysson (ARToolKit)
  Camera, processor, display together

AR Advertising

  Txt message to download AR application (200K)
  See virtual content popping out of real paper advert
  Tested May 2007 by Saatchi and Saatchi

2008 - Location Aware Phones

Motorola Droid Nokia Navigator

Real World Information Overlay
  Tag real world locations
  GPS + Compass input
  Overlay graphics data on live video

  Applications
  Travel guide, Advertising, etc
  Eg: Mobilizy Wikitude (www.mobilizy.com)
  Android based, Public API released
  Other companies
  Layar, AcrossAir, Tochnidot, RobotVision, etc

HIT Lab NZ Android AR Platform
  Architectural Application
  Loads 3D models
  a OBJ/MTL format
  Positions content in space
  GPS, compass
  Intuitive user interface
  toolkit to modify the model
  Connects to back end model database

Architecture
Web Interface

Add models

Web application java
and php server

Android

application

Database server

Postgres

History of Handheld and Mobile AR
1995 Handheld Display: NaviCam, AR-PAD, Transvision
1997 Wearable AR: Touring Machine, AR Quake
2001 Handheld AR – Thin Client: AR-PDA, Bat Portal
2003 Handheld AR – Self contained: Invisible Train
2003 Mobile Phone – 2D Vision: Mozzies, Symball
2003 Mobile Phone – Thin Client: ARphone
2004 Mobile Phone – Self contained: Moehring, Symbian

2012 State of the Art
Handheld Hardware available
PDA, mobile phones, external cameras
Sensors: GPS, accelerometer, compass
Software Tools are Available
Tracking: ARToolKitPlus, stbTracker, Vuforia
Graphics: OpenGL ES
Authoring: Layar, Wikitude, Metaio Creator
What is needed:
High level authoring tools
Content development tools
Novel interaction techniques
User evaluation and usability

Mobile AR Companies
  Mobile AR
  GPS + compass
  Many Companies
  Layar
  Wikitude
  Acrossair
  PressLite
  Yelp
  Robot vision
  Etc..

$2 million USD in 2010
$732 million USD in 2014

Handset Manufacturers
  Qualcomm
  $100 million USD investment
  Nokia
  25+ people in NRC
  Samsung
  Exploring the space
  Apple
  586 AR Applications on App Store
  Google
  Google goggles/Android AR Applications

Qualcomm

  Acquired Imagination
  October 2010 - Releases free Android AR SDK
  Computer vision tracking - marker, markerless
  Integrated with Unity 3D renderer
  http://developer.qualcomm.com/ar

Rock-em Sock-em

  Shared AR Demo
  Markerless tracking

Apple

  iPhone 4 SDK supports direct camera access
  Launches AR theme on App Store

Technology Components
  Software Platform
  Eg studierStube platform
  Developer Tools
  iOS, Android
  Tracking Technology
  Computer vision, sensor based
  Mobile Graphics
  OpenGL ES
  Interaction Methods
  Handheld Interaction

What are the challenges?
  Many platforms (operating systems)
  Slow CPUs (low clock rates, often no FPU)
  Difficulties with tracking
  Difficulties with visualizations that require a lot of data processing
  Small and slow memory access
  No or weak hardware 3D acceleration
  Difficulties with graphics
  Poor cameras
  Bad under low light, noise, etc

What makes a device interesting for AR?
  Open and easy to program
  Good camera
  Fast CPU, FPU is a plus
  Good H/W 3D support
  Large installed basis
  Easy access to devices
  GPS, accelerometer, compass
  Enough memory/storage

Typical Smart Phone Hardware
  CPU
  300-800+ Mhz
  GPU
  None, or Power VR Chip (OpenGL ES1.0/2.0)
  Input
  Touch screen, keyboard, keypad
  Sensors
  GPU, compass, accelerometer, camera (1.3-5mb+)
  Networking
  Bluetooth, Wifi, 3G
  Screen
  320x240 up to 800x480

HTC Desire
  Android
  Fast CPU (1GHz)
  Smaller screen
  3.7”, 800x480
  Multi-touch
  Hardware 3D
  GPS, compass and
accelerometer

iPhone 4
  Apple iOS 4.0
  Faster CPU (1.2GHz)
  High screen resolution
  3.5”, 960x640
  (Finally) camera API
  Multi-touch
  Hardware 3D
  GPS, compass, accelerometer
and gyroscope

Hardware Sensors
  Camera (resolution, fps)
  Maker based/markerless tracking
  Video overlap
  GPS (resolution, update rate)
  Outdoor location
  Compass
  Indoor/outdoor orientation
  Accelerometer
  Motion sensing, relative tilt

Studierstube ES Framework
  Typical AR application
framework
  Developed at TU Graz

End User
Application

Programming
Libraries

OS/Low Level API

Hardware

The Studierstube ES framework

User Interface - Application

Content

Graphics

Tracking

Platform

Mobile Development Environments
  Not like developing for desktops
  Wide range of different OS
  Limited CPU, low memory, poor graphics, no floating point
  Popular Mobile OS
  iPhone (Objective C)
  Android (Java, Native NDK wrapper)
  Windows Mobile (C#, C++, Visual Studio)
  Other
  Blackberry, Linux

Programming iPhone
  Harsh restrictions from Apple
  Apps have to go through the apps store
  Xcode IDE for development
  Nice development tools
  Objective-C
  Required for application development
  Can call into C/C++ code
  Limited camera API except in iOS 4
  Can overlay on live video
  No image processing

Android
  Hardware creators
  HTC, LG, Samsung, Motorola
  Widely available phone
  Different form factors – tablets, phones, PC, etc
  Multiple versions and fragmentation
  Android 1.0, 1.6
  Android 2.0, 2.1
  Free Tools
  Eclipse Development
  App Integrator

Computer Graphics on Mobile Phones
  Small screen, limited input options
  Limited support for accelerated graphics
  Most phones have no GPU
  Mobile Graphics Libraries
  OpenGL ES (1.0, 2.0)
-  Cross platform, subset of OpenGL
-  C/C++ low level library
  Java M3G
-  Mobile 3D graphics API for J2ME platform
-  Object importer, scene graph library
-  Support from all major phone manufacturers

OpenGL ES
  Small-footprint subset of OpenGL
  OpenGL is too large for embedded devices!
  Powerful, low-level API, full functionality for 3D games
  Can do almost everything OpenGL can
  Available on all key platforms
  Software and hardware implementations available
  Fully extensible
  Extensions like in OpenGL

OpenGL ES
SLIDE 61

OpenGL ES on mobile devices

OpenGL ES History
  Khronos Group – founded 2000
  Goal: create royalty free open multimedia standards
  Create OpenML specification
  Consortium of > 120 companies
  OpenGL ES working group – formed 2002
  20-30 companies
  Create API for 3D graphics on embedded devices
  Cleaned up and trimmed down version of OpenGL
  OpenGL ES 1.0 released 2004, 1.1 in 2005, 2.0 spec in 2007

Versions
  Two major tracks
  Not compatible, parallel rather than competitive
  OpenGL ES 1.x
  Fixed function pipeline
  Suitable for software implementations
  All 1.x are backwards compatible
  OpenGL ES 2.x
  Vertex and pixel shaders using GLSL ES
  All 2.x will be backwards compatible

Fixed Function (1.x)

h"p://www.khronos.org/opengles/2_X/

Programmable (2.x)

h"p://www.khronos.org/opengles/2_X/

Hardware supporting OpenGL ES 2.x
  Qualcomm SnapDragon
  Texas Instruments OMAP3
  NVIDIA Tegra (APX2500)
  Samsung S3C6410
  POWERVR SGX

  Now becoming available in phones!
  Typically come with OpenGL ES 1.x drivers too
  The emulation on GLES 2.0 hardware can be slow!

Mobile Augmented Reality’s goal
Create an affordable, massively multi-user, widespread platform

© Tinmith, U. of South Australia

Tracking Requirements for AR on Phones
  Fast and efficient
  Form factor: light and robust
  Track simultaneously
  a large number of objects
  by a large number of users
  Requires little or no …
  Device modification
  Manual calibration (targeting non-technical users)
  Instrumentation of the physical environment
  Low costs

How to not do it…

Ka-Ping Yee: Peephole Displays, CHI’03

Tracking on mobile phones
  Vision-based tracking
  Marker-based tracking
  Model-based natural feature tracking
  Natural feature tracking in unknown environments
  Sensor tracking
  GPS, inertial compass, gyroscope

Backpack-based

Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)

  Laptop, HMD
  Enhanced GPS (DGPS / RTK) + inertial sensor for viewpoint tracking
  Hand tracking w/ fiducial markers

Tablet PC / UMPC-based

Schall et al., 2006
  Hybrid tracking on UMPC
  Camera  fiducial marker tracking
  When no marker in view  inertial sensor + UWB tracking

PDA-based 1.

BatPortal (Newman et al., 2001) SHEEP (MacWilliams et al., 2003)
  PDA as thin client (rendering &
  Tracking by ART (external IR
tracking on server + VNC) cameras + retroreflective target)
  Ultrasonic tracking   Projection-based AR environ.

PDA-based 2.

Signpost on PDA (Wagner & Schmalstieg, 2003)

  First “truly” handheld AR platform: PDA + camera
  Standalone, self-contained AR system
  Optimized fiducial marker tracking library

History of non-AR Tracking on Phones (1)

AR-PDA (Gausemeier et al., 2003) Kick Real (Paelke et al., 2004)

  Model-based tracking   Edge detection of real foot + collision
  PDA = thin client detection w/ virtual ball
 tracking off-loaded to server   2D tracking and limited interaction
  Not real-time (tailored to game)

History of non-AR Tracking on Phones (2)

Mosquito Hunt (Siemens, 2003)
Marble Revolution (BitSide, 2004)
Pingis (VTT, 2006)
TinyMotion (Wang et al., 2006)
Game control w/ optical GUI control & input on cell phones
flow techniques w/ image differencing & block correlation

History of AR Tracking on Phones (1)
  2003
  ARToolKit on PDA
  Wagner et at.
  2004
  3D Marker on Phone
  Möhring et al.
  2005
  ARToolKit on Symbian
  Henrysson et al.

History of AR Tracking on Phones (2)

  2005
  Visual Codes
  Rohs et at.
  2008
  Advanced Marker Tracking
  Wagner et al.
  2008
  Natural Feature Tracking
  Wagner et al.

What can we do on today‘s mobile phones?
  Typical specs
  600+ MHz
  ~5MB of available RAM
  160x120 - 320x240 at 15-30 Hz camera
  Possible to do
  Marker tracking in 5-15ms
  Natural feature tracking in 20-50ms

Other Mobile Sensors
  Orientation
  Compass
  Relative movement/rotation
  Accelerometer, gyroscope
  Context
  Audio, light sensor, proximity
  Location
  GPS, A-GPS, Wifi positioning, Cell tower triangulation

Handheld HCI
  Consider your user
  Follow good HCI principles
  Adapt HCI guidelines for handhelds
  Design to device constraints
  Rapid prototyping
  User evaluation

Sample Handheld AR Interfaces
  Clean
  Large Video View
  Large Icons
  Text Overlay

Handheld Display vs Fixed Display

  Experiment comparing handheld moving, to handheld button input, small
fixed display, desktop display, large plasma
  Users performed (1) navigation task, (2) selection task
  Moving handheld display provided greater perceived FOV, higher degree of
presence, faster completion time

J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006

Outdoor AR: Limited Field of View

Possible solutions
  Overview + Detail
  spatial separation; two views

  Focus + Context
  merges both views into one view

  Zooming
  temporal separation

Zooming Views
  TU Graz – HIT Lab NZ - collaboration
  Zooming panorama
  Zooming Map

Zooming AR interfaces

Context
Compass
Zooming
Panorama
Zooming
Map

  Interface Types
  Compass (C)
  Compass + Zooming Panorama (CP)
  Compass + Zooming Map (CM)
  Compass, Zooming Panorama, Zooming Map (CPM)

Twitter 360

  www.twitter-360.com
  iPhone application
  See geo-located tweets in real world
  Twitter.com supports geo tagging

Wikitude – www.mobilizy.com

Blah
Blah Blah Blah
Blah Blah Blah
Blah
Blah
Blah
Blah Blah
Blah Blah Blah Blah
Blah Blah
Blah
Blah Blah Blah

Information Filtering
  Information Filtering (Julier et al. ’00)

•  Remove clutter by goal- and distance based filtering
•  User’s task is route finding: Sniper and relevant buildings are displayed;
objects, which are determined to be unnecessary, removed

HMD vs Handheld AR Interface
Wearable AR HandHeld AR

Output:
Display Input &
Output

Input

Handheld Interface Metaphors
  Tangible AR Lens Viewing
  Look through screen into AR scene
  Interact with screen to interact with AR
content
-  Eg Invisible Train

  Tangible AR Lens Manipulation
  Select AR object and attach to device
  Use the motion of the device as input
-  Eg AR Lego

Translation Study
Conditions
A: Object fixed to the phone (one handed)
B: Button and keypad input
C: Object fixed to the phone (bimanual)
- one hand for rotating tracking pattern

Results – Translation
•  9 subjects – within subject design
•  Timing
•  Tangible fastest
•  twice as fast as keypad

•  Survey
•  Tangible easiest (Q1)
•  Keypad most accurate (Q2)
•  Tangible quickest (Q3)
•  Tangible most enjoyable (Q4)

•  Ranking B
A
C
•  Tangible favored
Rank
1.44 2.56 2.0

Rotation Study
Conditions
A: Arcball
B: Keypad input for rotation about
the object axis
C: Object fixed to the phone (one handed)
D: Object fixed to the phone (bimanual)

Results – Rotation
•  Timing
•  Keypad(B) and Arcball(A) fastest

•  No significant survey
differences

A B C D
Rank 3.0 2.3 2.4 2.2

Design Guidelines
Apply handheld HCI guidelines for on-screen content
- large buttons, little text input, etc
Design physical + virtual interface elements
Pick appropriate interface metaphor
- “handheld lens” approach using handheld motion
- Tangible AR for AR overlay
Build prototypes
Continuously evaluate application

Glue Tracking - Markerless

  Search for “instant tracker”

Interface

  List View, Map View, AR View

Search.php
<?xml version="1.0" encoding="UTF-8"?>
<results>
<poi id="1" interactionfeedback="none">
<name><![CDATA[[Hotel Hello World]]]></name>
<description><![CDATA[[This is a beautiful, family hotel and restaurant, just around the
corner. Special Dinner and Rooms available.]]]></description>

<l>37.776685,-122.422771,0</l>
<mime-type>text/plain</mime-type>

<icon>http://dev.junaio.com/publisherDownload/tutorial/icon_map.png</icon>
<thumbnail>http://dev.junaio.com/publisherDownload/tutorial/thumb.jpg</thumbnail>
<phone>555/1234567</phone>
<homepage> http://www.hotelaroundthecorner.com </homepage>
</poi>
</results>

Mime Tag Types

  <mime-type>text/plain</mime-type>
  Changing mime tag changes media loaded

AR Browsers
  Commercial outdoor AR applications
  Junaio, Layar, Wikitude, etc
  All have their own language specifications
  Wikitude – ARML
  Junaio - XML
  Need for common standard
  Based on existing standards for geo-located content etc
  Support for dynamic/interactive content
  Easier to author mobile AR applications
  Easy to render on AR browsers

BirdsView - http://www.birdsview.de/

  Location Based CMS
  Add content, publish to Layar or Junaio

Limitations
  BirdsView Branding
  Their logo, not yours
  Limited POI Content
  Images, Text, URL
  Public Channel
  Your content + everyone else's

Hoppala Augmentation

  http://www.hoppala-agency.com/
  Rich media overlay

Metaio Creator

  Drag and drop Junaio authoring

3DOn

  Onsite Visualization Single Building
  GPS + Compass input
  Overlay graphics data on live video/Photos

426 Lecture 8: Mobile Augmented Reality

More Related Content

What's hot

Viewers also liked

Similar to 426 Lecture 8: Mobile Augmented Reality

More from Mark Billinghurst

Recently uploaded

426 Lecture 8: Mobile Augmented Reality