Wearable Computing - Part I: What is Wearable Computing?

Wearable Computing
Part I
What is wearable computing?
Daniel Roggen
2011

© Daniel Roggen www.danielroggen.net droggen@gmail.com
Naylor, G.: Modern hearing aids and future development trends, http://www.lifesci.sussex.ac.uk/home/Chris_Darwin/BSMS/Hearing%20Aids/Naylor.ppt

http://www.vuzix.com/consumer/products_wrap920ar.html

Mark Weiser: the visionary of ubiquitous computing
"The computer for the 21st century",
Scientific American, 1991
"Specialized elements of hardware and software, connected by
wires, radio waves and infrared, will be so ubiquitous that no one
will notice their presence."
"The most profound technologies are those that disappear. They
weave themselves into the fabric of everyday life until they are
indistinguishable from it."
"Ubiquitous computing in this context does not mean just
computers that can be carried to the beach, jungle or airport."
"My colleagues and I at PARC believe that what we call ubiquitous
computing will gradually emerge as the dominant mode of
computer access over the next 20 years."
1952 – 1992
Chief scientist at
Xerox PARC
invisible
in the background
enhancing the "reality" (not a virtual world!)

Motivated by continous technological progress

From Weiser to today
Weiser's SciAm paper (1991)
Pervasive computing
Pervasive: 2002
Ubiquitous computing
Ubicomp: 1999
Mobile computing
MobiSys: 2003
Wearable computing
ISWC: 1997
"Ambient Intelligence (AmI)"
Human-computer interaction
CHI: 1982
Sensor net
EWSN: 2004

The « founding fathers » of wearable computing
• MIT mid 1990s
• Steve Mann (Univ. Toronto)
– Humanistic Computing: “WearComp” as a
New Framework and Application for
Intelligent Signal Processing, Proc of the
IEEE 86(11), pp. 2123-2151, 1998
– Smart Clothing: The Shift to Wearable
Computing, Communications of the ACM,
39(8), pp. 23-24, 1996
• Thad Starner (Georgia Tech)
– Real-Time American Sign Language
Recognition Using Desk and Wearable
Computer Based Video, IEEE Trans on
Pattern Analysis and Machine Intelligence,
20(12), pp 1371-1375, 1998

“The watch long ago encountered many of the major issues confronting wearable
computing today. This paper […] discusses how the locations where the watch was
worn on the body have changed over time, examines a variety of user interfaces for
watches, and looks at how the watch affected cultural concepts of time and time
discipline.”
“The lessons for wearable computing are that the physical wearability will be
determined as much by fashion as by human anatomy, that the user interface will
gradually become simplified as people become more acquainted with computers, and
finally that the cultural impact will be a broadening of the definition of information, a
rationalization of representing information, and an increasing synchronization of
personal events.”
Martin, Time and Time Again: Parallels in the Development of the Watch and the Wearable
Computer. Proc. 6th International Symposium on Wearable Computers, 2002

http://www.research.ibm.com/WearableComputing/linuxwatch/linuxwatch.html
~2001

Another look behind
« It’s a digital electronic wristwatch, a personal calculator, an alarm clock, a
stopwatch, a timer, and a 200-year calendar, and its functions can interact to
produce previously unavailable results »
[Marion, Heinsen, Chin, Helmso. Wrist instrument opens new dimension in personal information, Hewlett-Packard Journal, 1977]

More than computing on the body: context-aware assistance
A modern definition of « wearable computing »
• Augmenting senses, cognition, communication
• Eminently personnal
• Continuously available
• Implicit interaction, usable despite cognitive load
• Adapts to the environment, to the user
• Proactive support
• At the right moment, with the right modality
Sense user's
context
Infer user's
needs
Provide
assistance
User reacts,
system adapts

Context awareness
• Merriam-Webster's Collegiate Dictionary:
– The word ``context'' is defined as ``the interrelated conditions in which
something exists or occurs
• Chen, Kotz:
– Context is the set of environmental states and settings that either
determines an application's behavior or in which an application event
occurs and is interesting to the user
• Dey (Understanding and Using Context, PUC, 2001):
– Context is any information that can be used to characterize the situation
of an entity. An entity is a person, place, or object that is considered
relevant to the interaction between a user and an application, including
the user and applications themselves
– A system is context-aware if it uses context to provide relevant
information and/or services to the user, where relevancy depends on the
user’s task
• Any knowledge about the user and his environment that enables
applications to be "smarter"

Physical Social
Mental
Mental
• Emotion awareness
– Own/Others
– Sadness, joy
– Depression
• Cognitive awareness
– Cognitive load
– Attention, concentration
– Stress
• …
Social
• Social interactions
– Detect known people
• Social network
– Information exchange
– Optimization of organizations
• Crowd / collective behavior
• …
Dimensions of context
Physical
• User location
– Absolute, relative
• User activity
– Manipulative gestures,
pointing movements, modes
of locomotion, posture,
composite and hierarchical
activities
• …
Environment
• Map of surrounding services
• Environment characteristics
– Temperature, light, humidity
• Radio fingerprints
• …
Environment
Most known so far

Mann, Smart Clothing: The Shift to Wearable
Computing, Comm. of the ACM, 1996
Roggen et al., Wearable Computing: Designing and Sharing
Activity-Recognition Systems Across Platforms, IEEE
Robotics&Automation Magazine, 2011
Campbell, A survey of mobile phone Sensing,
IEEE Communications Magazine, 2010

Nowadays, a typical wearable
• Sensors: to infer the user’s context
– On the body (wearable)
– In objects, environment ( « wearable »-« ambient » convergence)
• Wearable computer
– Mobile phone
• Context recognition
– Map sensor data to user context, needs
– User experience
• Feedback
– Display
– Vibrotactile
– Audio
• Integration in a Body/Personal Area Network
Sense user's
context
Infer user's
needs
Provide
assistance
User reacts,
system adapts

User
Experimental method
User studies
Interaction design / UX
HW
Textile engineering
Sensor
Communication
Processing
Healthcare
Sports
Entertainment
Industry
Assistedliving
Augmentedhuman
Multidisciplinary research
DSP
ML
AI
Reasoning
Modelling
Context/Activity awareness

Smart Assistants v.s. Data Logging
• Smart Assistants
– Context is recognized online / real-time
– Intelligence on body
– Sense and reacts to the user’s context
• Data Logging
– Data are locally stored or transmitted
– Offline analysis
– Often human intervention
– E.g. for medical purposes

Examples of wearable computing applications
1. Pervasive gaming
2. WearIT@Work: Supporting industrial workers with wearables
3. Parkinson's disease assistant: helping patients with freezing of gait
4. SMASH: sensing garment for rehabilitation

Pervasive gaming video

Industrial manufacturingIndustrial manufacturing
supported by wearablesupported by wearable
computingcomputing

Industrial Manufacturing Supported by On-body Computing
Wearable Computing in a work environment
European Union Project
23 Mio. €, 54 months
14 countries, 36 partners (Microsoft, SAP, EADS, Zeiss, HP, Siemens,
Tekniker, …)
Four pilot applications
Clinical ward round
Fire fighter
Aircraft maintenance
Car production
Stiefmeier et al., Wearable Activity Tracking in Car Manufacturing, IEEE Pervasive Computing Magazine, 2008

Training of Unskilled Workers
• Training process:
• 1st step: Theory
• 2nd step: Hands-on training of trainee
• Merge 1st and 2nd step with a wearable system capable of
activity tracking
1 2
• Despite automated production there remains a considerable amount
of manual production steps
• Training of unskilled workers is time consuming and costly

Case Study: Front Lamp Assembly
• "Real life" application of Wearable Computing
• Front lamp assembly for demonstration of
activity tracking
• Attachment parts: lamp, plastic bar, 3 screws
• Tools: 2 cordless screwdrivers, alignment tester
Klick
Klick

Sensing
Wearable sensors
– Inertial sensors [A]
– Force sensitive resistors [B]
– RFID reader [D]
– Ultrasonic receivers
– Using wireless links [E]
• Environmental sensors/infrastructure
– Mounted on car body
• Magnetic switches (Reed)
• Force sensitive resistors (FSR)
– Mounted on tools
• RFID-Tags [C]
– Ultrasonic beacons
~15 sensors are required for task tracking

Sensing

Sensor Calibration, Selection and Acquisition

Assembly task modeling and tracking
State0
State1
0 bar screws
State2
1 bar screw
State3
2 bar screws
State4
3 bar screws
State5
1 lamp scr.
State6
2 lamp scr.
State8
State11
State13
State7
State10
State9
State12
D1 / 0
/D1 / 1
A1*RF1*V / 2
A2*RF1*V / 3
A3*RF1*V / 4
A1*A2*RF1*V / 6
A1*A3*RF1*V / 7
A3*A3*RF1*V / 8
A1*A2*A3*RF1*V / 12
A4*RF2*V / 16
A5*RF2 / 17
A4*A5*RF2 / 19
/A1*/A2*/A3 / 5
/A1*/A2*A3 / 9
/A1*A2*/A3 / 10
A1*/A2*/A3 / 11
A1*A2*/A3 / 13
A1*/A2*A3 / 14
/A1*A2*A3 / 15
/A4*/A5 / 18
/A4*A5 / 20
A4*/A5 / 21
/A1*/A2*/A3 / 42
/A1*/A2*A3 / 43
/A1*A2*/A3 / 44
A1*/A2*/A3 / 45
/A1*/A2*/A3 / 46
/A4*/A5 / 47
/D2*D3*/D4 / 23
/D2*/D3*D4 / 24D2*/D3*/D4 / 22
D3*/D4 / 29 /D2*D3 / 36
D3 / 39
D2 / 40
D2*D3 / 37
D3*D4 / 31
D4 / 38
D2*D3*/D4 / 25
/D2*D3*D4 / 27
/D2*D4 / 33
D2*/D4 / 32
/D3*D4 / 30
D2*/D3 / 35
D2*D3*/D4 / 26
D2*D4 / 34
D2*D3*D4 / 28
/A4*/A5 / 50
/A4*/A5 / 53
/A4*/A5 / 49
/A4*/A5 / 48
/A4*/A5 / 51
/A4*/A5 / 52
/A4*/A5 / 41
State 3:
Screw B
tightened

Motion Jacket
• Jacket-integrated sensors
• 7 inertial sensor modules
• orientation data
• accelerometer and gyroscope data

Motion Model

• Contextual support
in the assembly
line
Stiefmeier et al., Wearable Activity Tracking in Car Manufacturing, IEEE Pervasive Computing Magazine, 2008

Benefits of Wearable Computing
• Medium-term
– Reduce training costs (time and manpower)
– Enable training at worker's own pace
– Increase efficiency ('productivity') during training
• Long-term
– Increase productivity of assembly
– Improve quality management (online documentation, work flow
monitoring)
– Increase security

Assistant for Parkinson’s disease patients with
freezing of gait
Funded by EC grant Nr FP6-018474-2
Bächlin et al. Wearable Assistant for Parkinson's Disease Patients With the Freezing of Gait Symptom, IEEE
Transactions on Information Technology in Biomedicine, 14(2), 2010

2nd
April 2009 36Daniel Roggen ETH Zürich / Wearable Computing Lab.
Parkinson disease patients with Freezing of Gait

2nd
Freezing of Gait (FOG)
• Parkinson’s disease (PD) is a common neurological disorder leading
to impaired motor skills
• When legs affected: FOG
– Developed by 50% of PD patients: 10% of those with mild syndrome, 80%
of severely affected [1]
• FOG is common cause of falls
[1] Macht et. al; Predictors of freezing in Parkinson’s Disease:
a survey of 6620 patients; Mov. Disord. 2007;22:953-6

2nd
State of the art
• Drug treatment (levodopa)
– FoG resistant to medication, difficult dosage, side effect, habituation
• Patients imagine...
– ... marching to command
– ... stepping over cracks in the floor
– or walk to music or a beat
• First studies with external rythmic cueing[2-5]
– Permanent cueing, in the laboratory
[2] Thaut et al. “Rhythmic auditory stimulation in gait training for parkinsons disease patients,”
Movement Disorders, vol. 11, no. 2, pp. 193–200, 1996.
[3] Rubinstein et al. “The power of cueing to circumvent dopamine deficits: a review of physical therapy
treatment of gait disturbances in parkinson’s disease,” Movement Disorders, vol. 17, no. 6, 2002.
[4] Lim et al. “Effects of external rhythmical cueing on gait in patients with parkinson’s disease: a
systematic review,” Clinical Rehabilitation, vol. 19, no. 7, pp. 695–713, 2005.
[5] van Wegen et al. “The effect of rhythmic somatosensory cueing on gait in patients with parkinson’s
disease,” Journal of the Neurological Sciences, vol. 248, no. 1-2, pp. 210–214, 2006.

2nd
Objective: context-aware external auditive cueing
• Provide auditive rhythmic cueing only when necessary
• Online FOG detection from body worn sensors
• Analyse patients' and physiotherapists' perspective

2nd
April 2009 40Daniel Roggen ETH Zürich / Wearable Computing Lab. 40
Wearable FOG assistant
• 3-D acceleration sensor
27x47x12mm3
; 22grams
fSR=64Hz, Bluetooth
Operation time 10h
• Wearable computer
Intel XScale processor
Linux system, CRN Toolbox[2]
132x82x30mm3
; 231grams,
Operation time ~6h
• Earphones
1Hz ticking sound
thigh sensor
shank sensor
trunk sensor
earphones
wearable
computer
] Bannach et. al; Distributed modular toolbox for multi-
modal context recognition. ARCS 2006; pp 99-113

2nd
FOG detection from acceleration signal
Power ratio in “freeze” and “locomotion” frequency range
Freeze band
Locomotor band

2nd
Online FOG detection: freeze/loco power ratio

2nd
FOG detection algorithm
Body motion
Acceleration Sensor
acc samples
Windowing +
Freq analysis
PDS
Energy sum
‘Loco’ band
Energy sum
‘Freeze’ band
Eloco Efreeze
Etotal FI*
Freeze index
thresholding
FOG detection
Total Energy
thresholding
FI
0/1

2nd
Protocol (~1h30/patient)
• Instruction on experiment
• 2 recording sessions
– without feedback (20mn)
– with feedback (20mn)
• Each session includes:
a) straight line walking

2nd
b) random walk

2nd
b) random walk
c) Activities of daily living
• Medication intake
• Debriefing with the therapist
• Questionnaire

2nd
Evaluation study

2nd
Data recordings evaluation
• 10 PD patients
– 7 males / 3 females
– 66.5 +- 4.8 years
– 2.7+- 0.6 H&Y scale
• 8h 20min of data recorded
• 8 out of 10 experienced FOG
• 235 FOG episodes occured

2nd
Results: Online detection accuracy
Overall:
• Sensitivity = 73.1%
• Specificity = 81.6%

2nd
Video: Feedback ON

2nd
Conclusion
• First study with online FOG detection & cueing
• Sensitivity 73.1%; Specificity 81.6% (global parameters)
• Promising feedback from patients and therapists
• PD patient may benefit from context-aware cueing
• Only 10 patients - subjective reports must be taken carefully
• Results support conducting a larger scale study, eventually outside of
laboratory

Horizontal elongation Vertical elongation
Lifting shoulders
A Method to Measure Elongations of Clothing, C. Mattmann, T. Kirstein, G. Tröster. Proc. Ambience 05, 1st International
Conference on Intelligent Ambience and Well-Being, Tampere, Finland, September 19-20 2005
Design Concept of Clothing Recognizing Back Postures; C. Mattmann, G. Tröster; Proc. 3rd IEEE-EMBS International Summer
School and Symposium on Medical Devices and Biosensors (ISSS-MDBS 2006), Boston, September 4-6, 2006.
sensors
Backmanager
• Garment measuring upper body posture with strain sensors
• Detection of "unhealthy" postures
• Corrective feedback to the user

Backmanager
Design Concept of Clothing Recognizing Back Postures; C. Mattmann, G. Tröster; Proc. 3rd IEEE-EMBS International Summer School and Symposium on
Medical Devices and Biosensors (ISSS-MDBS 2006), Boston, September 4-6, 2006.

SMASH: garment for posture sensing
• Applications:
– Practicing rehabilitation movements
– Posture reminders
Harms et al., Rapid prototyping of smart garments for activity-aware applications. Journal of Ambient Intelligence
and Smart Environments, 2009

SMASH: garment for posture sensing
Harms et al., Rapid prototyping of smart garments for activity-aware applications. Journal of Ambient Intelligence
and Smart Environments, 2009
Postures recognition for shoulder rehabilitation Recognition of back angle for posture reminder

Commercial potential or toy research?
Amft&Lukowicz, From Backpacks to Smartphones: Past, Present, and Future of Wearable Computers, IEEE Pervasive Computing Magazine, 2009
Zypad
http://www.zypad.com
WT4000 Wearable Terminal
Motorola
Guinness World Records
“fastest-selling consumer device”
133’333 units/day between
November and January
Nike+ SportBand

• Location-based reminders
– E.g. « GeoMemo App » for Android
• « Bump App » for iPhone/Android
– Bump phones together to share data between
phones
– Example of motion recognition
• Place recommendations
– E.g. foursquare
– Location-awareness + friend network
Context aware applications

Multidisciplinary
• Wearable computing
• Robotics, Sensors, IT, Networks
• Medicine
• (Neuro-)Psychology
• Cognitive science, Social sciences
Shift towards richer "human-like" devices or smart assistants.
They know what we need, when, and how we want it.
Better awareness of the user's needs, state and internal
motivations (compared to personal computing).
Applications / Users
• Healthcare
• Social / behavioral sciences
• Psychology
• Gaming
• HCI
• Robotics
Some last remarks….
Adapts and reacts to the user’s needs

For further reading
Founding fathers and background
• Weiser, Computer for the 21st Century, IEEE PCM, reprint, 2002 (original: Scientific American, 1991)
• Weiser, Hot Topics - Ubiquitous computing, IEEE Computer, 1993
• Mann, Wearable Computing as Means for Personal Empowerment, ISWC, 1998
• Mann, Humanistic Computing-WearComp as a New Framework and Application for Intelligent Signal Processing
• Mann, Smart Clothing-The Shift to Wearable Computing, Communications of the ACM, 1996)
• Starner, Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video, IEEE Trans on Pattern
Analysis and Machine Intelligence, 20(12), pp 1371-1375, 1998
Looking back
• Bush, As We May Think, The Atlantic Monthly, 1945
• Marion, Heinsen, Chin, Helmso. Wrist instrument opens new dimension in personal information, Hewlett-Packard Journal, 1977
• Martin, Time and Time Again: Parallels in the Development of the Watch and the Wearable Computer. Proc. 6th International Symposium
on Wearable Computers, 2002
• Thorp, The Invention of the First Wearable Computer, IEEE Int.Symp. on Wearable Computer 1998
Context and activity-aware computing
• Dey, Understanding and Using Context, Personal and Ubiquitous Computing,2001
• Lukowicz et al., On-Body Sensing: From Gesture-Based Input to Activity-Driven Interaction, IEEE Computer, 43(10), pp. 92-96, 2010
• N. Davies, D. Siewiorek, R. Sukthankar, Special Issue: Activity-Based Computing, IEEE Pervasive Computing, 7(2), pp. 20-21, 2008
Challenges
• Starner, The Challenges of Wearable Computing-Part 1&2, IEEE Pervasive Computing Magazine, 2001
Applications
• Stiefmeier et al., Wearable Activity Tracking in Car Manufacturing, IEEE Pervasive Computing Magazine, 2008
• Bächlin et al. Wearable Assistant for Parkinson's Disease Patients With the Freezing of Gait Symptom, IEEE Transactions on Information
Technology in Biomedicine, 14(2), 2010
• Harms et al., Rapid prototyping of smart garments for activity-aware applications. Journal of Ambient Intelligence and Smart
Environments, 2009
• Paradiso, Gips, Laibowitz, Sadi, Merrill, Aylward, Maes, Pentland, Identifying and facilitating social interaction with a wearable wireless
sensor network, Personal and Ubiquitous Computing, 4(2), pp.137-152, 2010
• Pentland, Healthwear: Medical Technology Becomes Wearable, Computer, 37(5), pp. 42-49, 2004
• Eagle, Pentland, Lazer, Inferring friendship network structure by using mobile phone data, Proc Natl Acad Sci, 2009
Mobile phones
• Lane, Miluzzo, Lu, Peebles, Choudhury, Campbell, A Survey of Mobile Phone Sensing, IEEE Communications Magazine, 48(9), pp. 140-
180, 2010

Wearable Computing - Part I: What is Wearable Computing?

More Related Content

What's hot

Viewers also liked

Similar to Wearable Computing - Part I: What is Wearable Computing?

More from Daniel Roggen

Recently uploaded

Wearable Computing - Part I: What is Wearable Computing?