Download as PDF, PPTX
![Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-11-320.jpg)
![Early Examples
• Interaction without devices:
• BodySpace [Strachan 2007]: Functions to body position
• Abracadabra [Harrison 2007]: Magnets on finger tips
• GesturePad [Rekimoto 2001]: Capacitive sensing in clothing
• Palm-based Interaction
• Haptic Hand [Kohli 2005]: Using non-dominant hand in VR
• Sixth Sense [Mistry 2009]: Projection on hand
• Brainy Hand [Tamaki 2009]: Head worn projector/camera](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-14-320.jpg)
![ImaginaryPhone
• Gustafson, S., Holz, C., & Baudisch, P. [2011]](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-15-320.jpg)
![Invisible Interfaces – Gestures in Space
• Gustafson, S., Bierwirth, D., & Baudisch, P. [2010]
• Using a non-dominant hand stabilized interface.](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-17-320.jpg)
![Class Extending
• Take one class and add functionality to it with a new class
• Eg enemy class for video game
!class Enemy {!
! !int x, y; //position!
! !.. .. !
! !public void draw() {!
! !//draw my picture to the screen at the proper location }!
!}!
• Want to draw enemy twice – create new class
!//on a "DoubleEnemy.h" file!
!class DoubleEnemy: public Enemy // class[className]:[privacy][extended Class]{}!
!{!
! !public void draw();//the actual code inthe "DoubleEnemy.cpp" file!
!}; // note the ";" at the end of the class statement!](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/85/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-48-320.jpg)
![Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real andVirtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/75/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-11-2048.jpg)
![Early Examples
• Interaction without devices:
• BodySpace [Strachan 2007]: Functions to body position
• Abracadabra [Harrison 2007]: Magnets on finger tips
• GesturePad [Rekimoto 2001]: Capacitive sensing in clothing
• Palm-based Interaction
• Haptic Hand [Kohli 2005]: Using non-dominant hand in VR
• Sixth Sense [Mistry 2009]: Projection on hand
• Brainy Hand [Tamaki 2009]: Head worn projector/camera](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/75/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-14-2048.jpg)
![ImaginaryPhone
• Gustafson, S., Holz, C., & Baudisch, P. [2011]](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/75/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-15-2048.jpg)
![Invisible Interfaces – Gestures in Space
• Gustafson, S., Bierwirth, D., & Baudisch, P. [2010]
• Using a non-dominant hand stabilized interface.](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/75/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-17-2048.jpg)
![Class Extending
• Take one class and add functionality to it with a new class
• Eg enemy class for video game
!class Enemy {!
! !int x, y; //position!
! !.. .. !
! !public void draw() {!
! !//draw my picture to the screen at the proper location }!
!}!
• Want to draw enemy twice – create new class
!//on a "DoubleEnemy.h" file!
!class DoubleEnemy: public Enemy // class[className]:[privacy][extended Class]{}!
!{!
! !public void draw();//the actual code inthe "DoubleEnemy.cpp" file!
!}; // note the ";" at the end of the class statement!](https://image.slidesharecdn.com/lecture5-openframeworksfinal-160827070744/75/COMP-4026-Lecture-5-OpenFrameworks-and-Soli-48-2048.jpg)
Uploaded byMark Billinghurst
COMP 4026 Lecture 5 OpenFrameworks and Soli
This document outlines a lecture on advanced human-computer interaction technologies, focusing on wearable computing, augmented reality, virtual reality, and the Project Soli radar-based motion sensing technology. It details the process for a class project involving advanced technology design and covers OpenFrameworks as a toolkit for creative coding in C++. Key concepts discussed include the capabilities of wearable computers, the definition of augmented reality, and the programming nuances of C++ related to OpenFrameworks.
More Related Content
Similar to COMP 4026 Lecture 5 OpenFrameworks and Soli
COMP 4026 Lecture 5 OpenFrameworks and Soli
- 1. LECTURE 5: OPENFRAMEWORKS ANDSOLI COMP 4026 – Advanced HCI Semester 5 - 2016 Mark Billinghurst University of South Australia August 25th 2016
- 2.
- 3. Advanced Interface Technology • WearableComputers • Augmented Reality • Virtual Reality • Invisible Interfaces • Environment Sensing • Physiological Sensing
- 4. Class Project 1. PickAdvanced Technology 2. Brainstorm use case 3. Develop conceptual design 4. Prototype interface/experience design 5. Conduct user evaluation 6. Repeat steps 3-5 7. Write report
- 5. Wearable Computing ▪ Computeron the body that is: ▪ Always on ▪ Always accessible ▪ Always connected ▪ Other attributes ▪ Augmenting user actions ▪ Aware of user and surroundings
- 6.
- 7.
- 9.
- 10.
- 11. Augmented Reality Definition • DefiningCharacteristics [Azuma 97] • Combines Real andVirtual Images • Both can be seen at the same time • Interactive in real-time • The virtual content can be interacted with • Registered in 3D • Virtual objects appear fixed in space Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.
- 12. Virtual Reality • ImmersiveVR • Headmounted display, gloves • Separation from the real world
- 13.
- 14. Early Examples • Interactionwithout devices: • BodySpace [Strachan 2007]: Functions to body position • Abracadabra [Harrison 2007]: Magnets on finger tips • GesturePad [Rekimoto 2001]: Capacitive sensing in clothing • Palm-based Interaction • Haptic Hand [Kohli 2005]: Using non-dominant hand in VR • Sixth Sense [Mistry 2009]: Projection on hand • Brainy Hand [Tamaki 2009]: Head worn projector/camera
- 15. ImaginaryPhone • Gustafson, S.,Holz, C., & Baudisch, P. [2011]
- 16.
- 17. Invisible Interfaces –Gestures in Space • Gustafson, S., Bierwirth, D., & Baudisch, P. [2010] • Using a non-dominant hand stabilized interface.
- 18. Project Soli • UsingRadar to support free-hand spatial input
- 19. Google Tango • Tablet basedsystem • Android OS • Multiple sensors • RGBD Sensor • IR Structured light • Inertial sensors • High end graphics • Nvidia tegra chip
- 20. Physiological Sensors • Sensing userstate • Body worn devices • Multiple possible sensors • Physical activity • Eye tracking, gaze • Heart rate • GSR • Breathing • Etc
- 21. Tobii Eye Tracker • Wearableeye tracking system • Natural data capture • Scene camera capture • Recording/streaming eye gaze, 60 Hz sampling
- 22.
- 23. OpenFrameworks (www.openframeworks.cc) • Open sourcetoolkit designed for creative coding • Developed by Z. Lieberman,T.Watson and A. Castro • Framework – collection of libraries • Written in C++ • More powerful than Processing, but more complicated • Must use IDE for development • Xcode,Visual Studio, Code::Blocks • Runs on Mac,Windows, Linux platforms
- 26. Why use oFinstead of Processing • Speed • Accessibility of low level information • Debugger • C++ • Version control • Cross Platform
- 27. OpenFrameworks vs.Processing • Makingproject visible on Internet - Processing • Make a project with lots of 3D graphics - OpenFrameworks • Make a project for lots of different computers/OS – Processing • Make a project using an external library like the OpenCV computer vision library – OpenFrameworks • Make a project that interfaces with the Arduino board - Either
- 28. OpenFrameworks Installation • addons:added libraries from user community. Must be explicitly included in programs using them • apps: store your programs here.Also contains example code. • libs: where the core libraries of OpenFrameworks are stored. Also contains core openFrameworks folder
- 31.
- 40.
- 41.
- 44. Typical FunctionTypes • setup( ) • loadassets • Initialize values • Initialize addons or components • update( ) • calculations • increment video frames • draw( ) • draw shapes/images/videos • use GLSL Shaders
- 47. Classes in C++ • C++ classes comprise of two files. It helps to think of these two files as a recipe. • The header file (.h) is like the list of ingredients, and contains: • Any preprocessor statements there to prevent multiple header definitions • Any include statements to other classes • Any class extension statements • Any variables local to the class • Prototypes of any functions to be contained in the class • Security settings of these functions and variables (e.g. public, private, protected, etc). • and a body file (.cpp) which is like the instructions on what to do with the ingredients and contains: • An include statement that references the .h file • All of the code to fill in the function prototypes.
- 48. Class Extending • Takeone class and add functionality to it with a new class • Eg enemy class for video game !class Enemy {! ! !int x, y; //position! ! !.. .. ! ! !public void draw() {! ! !//draw my picture to the screen at the proper location }! !}! • Want to draw enemy twice – create new class !//on a "DoubleEnemy.h" file! !class DoubleEnemy: public Enemy // class[className]:[privacy][extended Class]{}! !{! ! !public void draw();//the actual code inthe "DoubleEnemy.cpp" file! !}; // note the ";" at the end of the class statement!
- 49. Pass byValue vs.byReference • void functn(int num) – pass by value • void functn(<class> test) – pass by reference • sends address of where class stored • use pointers to pass arrays back and forth through functions int num = 5; value stores address of variable value
- 50. void setup() { int num= 1; addOne(num); print(num); } void addOne(int num) { num++; } class Test { int num=0; } void setup() { Test test = new Test(); test.num=1; addOne(test); print(test.num); } void addOne(Test test) { test.num++; } Pass by Value Pass by Reference
- 51. & and * • In C++ you need to explicitly state whether you are passing something by value or by reference. • Use & (referencing) and * (dereferencing) symbols • the & symbol is used to acquire the memory address of a variable or function b=1;! a = &b; // a now equal to memory address of b! a++; // memory address of b + 1! *a++; // value a +1 (increments b as well)!
- 52. Example • What doesthis code do? ! ! !int x;! ! ! !int *ptr;! ! ! ! !x=5;! ! ! !ptr = &x;! ! ! !*ptr = 10;!
- 53. 2D Image Functions • Colors • ofFill(); • ofCircle(100,400,80); • ofSetHexColor(0x000000); ofSetColor(255,0,0,127); • Primitives • ofCircle(100,400,80); • ofRect(400,350,100,100); • ofLine(600,300,800, 250); • ofDrawBitmapString("rectangles", 275,500);
- 54.
- 55. Circle Grid • Setting thesize of the window. • Processing: • size(800, 600, OPENGL); • openFrameworks: • ofSetupOpenGL(&window, 800, 600, OF_WINDOW); • function is called in main() in the file main.cpp.
- 56. Circle Grid • DeclaringVariables • Processing: • Declare thevariables you need right after you import the libraries you need. • openFrameworks: • Declare variables in the file testApp.h, after the line void windowResized(int w, int h);.
- 57. Circle Grid • Background Color • Processing: • background(0); will set the background of your sketch to black. You need to call the function inside draw() to draw the background each frame. • openFrameworks: • Call ofBackground(0, 0, 0); once inside the setup() method. openFrameworks will draw the background automatically each frame.You can disable this by calling ofSetBackgroundAuto(false) within setup() in the file testApp.cpp.
- 58. Circle Grid • DrawingCircles • Processing: • after you have set the stroke and fill, use ellipse(50, 50, 20, 20); to draw a circle with a diameter of 20 at (50, 50). • openFrameworks: • you can use ofCircle(50, 50, 10); to draw the same circle.You could also use ofEllipse(50, 50, 20, 20);. If you want to draw a circle with a stroke you will need to call the function to draw the circle two times. Once for the fill and once for the stroke. ofSetColor(255, 255, 255);! ! !ofFill();! ! !ofCircle(50, 50, 20);!
- 59. Graphics Demo -graphicsExample • setup( ) method • draw( ) method
- 60. Drawing Polygons • Mustbegin and end a shape • ofVertex, ofCurveVertex, ofBezierVertex !ofBeginShape();! ! ! !ofVertex(200,135);! ! ! !ofVertex(15,135);! ! ! !ofVertex(165,25);! ! ! !ofVertex(105,200);! ! ! !ofVertex(50,25);! !ofEndShape();! •
- 61.
- 62.
- 63. Importing Libraries • Largeset of oF addon libraries (> 450) • http://ofxaddons.com/ • Just download library to addons directory, then include library #include “myLibrary.h”! • Sample libraries • ofxOpenCv • ofxVectorGraphics • ofxVectorMath • ofxNetwork • ofvOsc
- 64. Examples • VectorGraphicsExample • 3DModelLoaderExample • Loading 3D models • assimpleExample • 3Danimation • openCVExample • Hand segmentation
- 65.
- 66.
- 67.
- 68. Nike + PaintWith Your Feet +openFrameworks +GPS
- 69.
- 70.
- 71. Overview • Soli usesradar to detect fine scale finger motion
- 72.
- 73.
- 74. Radar Fundamentals • Radartracks moving objects • Measures response to Radar waves sent from transmitter
- 75. Radar Reflections fromHand • Multiple reflection points
- 76. Signal Processing • Signalreceived combination of slow and fast time
- 77.
- 78. Processing Pipeline • Fromraw hand motion to recognized gestures
- 79.
- 80. Signs vs. Actions • Soli recognizes hand actions
- 81. Virtual Tools • Usevirtual tool metaphor • Change with distance
- 82. Types of VirtualTools • asdasf
- 83. Basic Gesture Movement • Easily recognize distinctive gesture motions
- 84. • > 90%accuracy on filtered results (Bayesian Filter) Recognition Results
- 85. Applications • Gesture interactionwith objects • Smart watch, car console • Gesture interaction with environment • Furniture, walls • Other applications • Material recognition, Gaming, Object scanning
- 86.
- 87.
- 88.
- 89. Future Research • Radar Sensing • Radar clutter, multi-path reflections, occlusion, etc • Machine Learning • New gesture recognition approaches • Human Factors • Measuring human performance abilities, requirements • Interaction Design • New interaction modalities, metaphors
- 90. Background Reading Lien, Jaime,Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. "Soli: ubiquitous gesture sensing with millimeter wave radar." ACM Transactions on Graphics (TOG) 35, no. 4 (2016): 142.
- 91.