Download as PDF, PPTX
Uploaded byMark Billinghurst
Comp4010 lecture6 Prototyping
The document discusses various types of augmented reality (AR) interactions, including browsing interfaces, 3D AR, tangible AR, and natural interfaces that facilitate user engagement through gestures, speech, and gaze. It emphasizes the importance of prototyping in interaction design, outlining various high and low fidelity techniques, tools, and processes for developing AR applications. Case studies illustrate the application of AR in navigational aids and immersive experiences, alongside tools for digital authoring and interactive design.
More Related Content
Similar to Comp4010 lecture6 Prototyping
Comp4010 lecture6 Prototyping
- 1. PROTOTYPING AND DESIGN COMP4010 Lecture Six Mark Billinghurst August 31st 2021 mark.billinghurst@unisa.edu.au
- 2.
- 3.
- 4. Different Types ofAR Interaction 1. Browsing Interfaces • simple (conceptually!), unobtrusive 2. 3D AR Interfaces • expressive, creative, require attention 3. Tangible Interfaces • embedded into conventional environments 4. Tangible AR • combines TUI input + AR display 5. Natural AR Interfaces • interact with gesture, speech and gaze
- 5. 1. AR Interfacesas Data Browsers • 2D/3D virtual objects are registered in 3D • “VR in Real World” • Interaction • 2D/3D virtual viewpoint control • Applications • Visualization, training
- 6. 2. 3D ARInterfaces • Virtual objects displayed in 3D physical space and manipulated • HMDs and 6DOF head-tracking • 6DOF hand trackers for input • Interaction • Viewpoint control • Traditional 3D user interface interaction: manipulation, selection, etc. Kiyokawa, et al. 2000
- 7. 3. Augmented Surfacesand Tangible Interfaces • Basic principles • Virtual images are projected on a surface • Physical objects are used as controls for virtual objects • Support for collaboration Wellner, P. (1993). Interacting with paper on the DigitalDesk. Communications of the ACM, 36(7), 87-96.
- 8. Tangible User Interfaces(Ishii 97) • Create digital shadows for physical objects • Foreground • graspable UI • Background • ambient interfaces
- 9. Tangible Interface: ARgroove •Collaborative Instrument • Exploring Physically Based Interaction • Map physical actions to Midi output • Translation, rotation • Tilt, shake
- 10. 4. Tangible AR:Back to the Real World • AR overcomes display limitation of TUIs • enhance display possibilities • merge task/display space • provide public and private views • TUI + AR = Tangible AR • Apply TUI methods to AR interface design Billinghurst, M., Kato, H., & Poupyrev, I. (2008). Tangible augmented reality. ACM Siggraph Asia, 7(2), 1-10.
- 11. Tangible AR: Tiles(Space Multiplexed) • Tiles semantics • data tiles • operation tiles • Operation on tiles • proximity • spatial arrangements • space-multiplexed Poupyrev, I., Tan, D. S., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001, July). Tiles: A Mixed Reality Authoring Interface. In Interact (Vol. 1, pp. 334-341).
- 12. Tangible AR: Time-multiplexedInteraction • Use of natural physical object manipulations to control virtual objects • VOMAR Demo • Catalog book: • Turn over the page • Paddle operation: • Push, shake, incline, hit, scoop Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000, October). Virtual object manipulation on a table-top AR environment. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000) (pp. 111-119). IEEE.
- 13. 5. Natural ARInterfaces • Goal: • Interact with AR content the same way we interact in the real world • Using natural user input • Body motion • Gesture • Gaze • Speech • Input recognition • Nature gestures, gaze • Multimodal input FingARtips (2004) Tinmith (2001)
- 14. External Fixed Cameras •Overhead depth sensing camera • Capture real time hand model • Create point cloud model • Overlay graphics on AR view • Perform gesture interaction Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in space: Gesture interaction with augmented-reality interfaces. IEEE computer graphics and applications, 34(1), 77-80.
- 15. Head Mounted Cameras •Attach cameras/depth sensor to HMD • Connect to high end PC • Computer vision capture/processing on PC • Perform tracking/gesture recognition on PC • Use custom tracking hardware • Leap Motion (Structured IR) • Intel RealSense (Stereo depth) Project NorthStar (2018) Meta2 (2016)
- 16. Self Contained Systems •Sensors and processors on device • Fully mobile • Customized hardware/software • Example: Hololens 2 (2019) • 3D hand tracking • 21 points/hand tracked • Gesture driven interface • Constrained set of gestures • Multimodal input (gesture, gaze, speech)
- 17. Speech Input • Reliablespeech recognition • Windows speech, Watson, etc. • Indirect input with AR content • No need for gesture • Match with gaze/head pointing • Look to select target • Good for Quantitative input • Numbers, text, etc. • Keyword trigger • “select”, ”hey cortana”, etc https://www.youtube.com/watch?v=eHMkOpNUtR8
- 18. Eye Tracking Interfaces •Use IR light to find gaze direction • IR sources + cameras in HMD • Support implicit input • Always look before interact • Natural pointing input • Multimodal Input • Combine with gesture/speech Camera IR light IR view Processed image Hololens 2
- 19. Evolution of ARInterfaces Tangible AR Tangible input AR overlay Direct interaction Natural AR Freehand gesture Speech, gaze Tangible UI Augmented surfaces Object interaction Familiar controllers Indirect interaction 3D AR 3D UI Dedicated controllers Custom devices Browsing Simple input Viewpoint control Expressiveness, Intuitiveness
- 20.
- 21. Typical Interaction DesignCycle Develop alternative prototypes/concepts and compare them, And iterate, iterate, iterate....
- 22. Prototyping in InteractionDesign Key Prototyping Steps
- 23.
- 24. Tom Chi’s PrototypingRules 1. Find the quickest path to experience 2. Doing is the best kind of thinking 3. Use materials that move at the speed of thought to maximize your rate of learning
- 25. How can wequickly prototype XR experiences with little or no coding?
- 26. From Idea toProduct
- 27. XR Prototyping Techniques Lo- Fi Hi- Fi Easy Hard Digital Authoring Immersive Authoring Web-Based Development* Cross-Platform Development* Native Development* *requires scripting and 3D programming skills Sketching Paper Prototyping Video Prototyping Wireframing Bodystorming Wizard of Oz
- 28. XR Prototyping Tools LowFidelity (Concept, visual design) • Sketching • Photoshop • PowerPoint • Video High Fidelity (Interaction, experience design) • Interactive sketching • Desktop & on-device authoring • Immersive authoring & visual scripting • XR development toolkits
- 30.
- 31. Storyboarding - Describingthe Experience http://monamishra.com/projects/Argo
- 32. Wireframes It’s about - Functionalspecs - Navigation and interaction - Functionality and layout - How interface elements work together - Defining the interaction flow/experience Leaving room for the design to be created
- 33. Mockup It’s about - Lookand feel - Building on wireframe - High fidelity visuals - Putting together final assets - Getting feedback on design
- 34. Interface Sketching inVR Using VR applications for rapid prototyping - Intuitive sketching in immersive space - Creating/testing at 1:1 scale - Rapid UI design/layout Examples - Quill - https://quill.fb.com/ - Tilt Brush - https://www.tiltbrush.com/
- 35. Microsoft Maquette •Prototype AR/VRinterfaces from inside VR •3D UI for spatial prototyping •Bring content into Unity with plug-in •Javascript support
- 36. Scene Assembly InAR • Many tools for creating AR scenes • Drag and drop your assets • Develop on web, publish to mobile • Examples • Catchoom - CraftAR • Blippar - Blipbuilder • ARloopa - Arloopa studio • Wikitude - Wikitude studio • Zappar - ZapWorks Designer
- 37.
- 38. Video Sketching • Process •Capture elements of real world • Use series of still photos/sketches in a movie format. • Act out using the product • Benefits • Demonstrates the product experience • Discover where concept needs fleshing out. • Communicate experience and interface • You can use whatever tools you want, e.g. iMovie.
- 39. Tvori - AnimatingAR/VR Interfaces • Animation based AR/VR prototyping tool • https://tvori.co/ • Key features • Model input, animation, etc • Export 360 images and video • Simulate AR views • Multi-user support • Present in VR • Create VR executable
- 40.
- 41. MOBILE AUGMENTED REALITY FORSPATIAL NAVIGATION Sharon Brosnan 0651869 Bachelor of Science in Digital Media Design
- 42.
- 43. Project Goals Make amobile AR Navigational Aid ○ Easy to use, clear and understandable ○ Useful to visitors ○ Creating interaction between the visitor and the aid through the user interface ○ Engage the visitor To ensure this… ○ It is necessary to understand the visitor of a navigational aid in Bunratty Folk Park ○ Identify visitor motives and goals while going through the Folk Park.
- 44. NeedsAnalysis Understanding Technology andRelated Work • Literature • Similar Technologies • Electronic Tours in Museum Settings • Interactions design in Outdoor Museums Understanding the User over time • Observations • Interviews
- 45. Findings • Most visitorsdo not use the map • Most visitors have mobile phones • Visitors want more information • View the Folk Park at their own pace • Information should be straight to the point • Large social interaction within groups Needs Analysis
- 46. Desired Functionality • ViewOptions • CameraView • MapView • ListView • Sub-Options • Places • Events • Restaurants • Augmented Reality Features for navigation • Text Information • 3D Objects • 3D Tour Guide • 3D Placement of Buildings
- 47.
- 48. ITERATIVE DESIGN PROCESS Prototypingand UserTesting • Low Fidelity Prototyping • Sketches • Paper Prototyping • Post-It Prototyping • PowerPoint Prototyping • High Fidelity Prototyping • Wikitude
- 49. INITIAL SKETCHES (Photoshop) Pros: •Good for idea generation • Cheap • Concepts seem feasible Cons: • Not great feedback gained • Photoshop not fast enough for making changes
- 50.
- 51. POWERPOINT PROTOTYPING Benefits • Usedfor User Testing • Interactive • Functionalities work when following the story of Scenario 1 • Quick • Easy arrangement of slides User Testing • Participants found • 15 minute sessions screen captured • ‘Talk Allowed’ technique used • Notes taken • Post-Interview
- 52. WIKITUDE • Popular augmentedreality browser for mobile devices • Mapping • Point of Interest abilities • Multiplatform • Shows the points of interest of Bunratty Folk Park • Markers can be selected in and an information pop-up appears
- 53. VIDEO PROTOTYPE ¢ Flexibletool for capturing the use of an interface ¢ Elaborate simulation of how the navigational aid will work ¢ Does not need to be realistic in every detail ¢ Gives a good idea of how the finished system will work
- 54.
- 55.
- 56. XR Prototyping Tools LowFidelity (Concept, visual design) • Sketching • Photoshop • PowerPoint • Video High Fidelity (Interaction, experience design) • Interactive sketching • Desktop & on-device authoring • Immersive authoring & visual scripting • XR development toolkits
- 57. XR Prototyping Techniques Lo- Fi Hi- Fi Easy Hard Digital Authoring Immersive Authoring Web-Based Development* Cross-Platform Development* Native Development* *requires scripting and 3D programming skills Sketching Paper Prototyping Video Prototyping Wireframing Bodystorming Wizard of Oz
- 58. Interactive Sketching •Pop App ●Pop - https://marvelapp.com/pop ● Combining sketching and interactivity on mobiles ● Take pictures of sketches, link pictures together
- 59.
- 60.
- 61. Proto.io • Web basedprototyping tool • Visual drag and drop interface • Rich transitions • Scroll, swipe, buttons, etc • Deploy on device • mobile, PC, browser • Ideal for mobile interfaces • iOS, Android template • For low and high fidelity prototypes
- 62.
- 63. Case Study: CityViewAR(2021)
- 64.
- 65. CityViewAR Application Goal: Createa mobile app that allowed people to view immersive panoramas on site, showing the 2011 damage Key technology: Mobile phone, GPS, 360 panorama, map User Experience: People walking the city streets will be able to be immersed in the earthquake damage from 10 years ago
- 66. CityViewAR Prototyping Process 1.Sketch Interaction Flow (Paper) 2. Create Wireframe (Powerpoint) 3. Sketch Screens (Paper) 4. Test Transitions (Pop) 5. Collect/Create Assets (Photoshop, various) 6. Create High Fidelity Prototype (Proto.io) 7. Code (Unity, Mapbox) 8. Make video demo (Various) 9. Publish (Android, iOS)
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73. High Fidelity -Proto.io
- 74.
- 75.
- 76.
- 77. Deployment to AppStore
- 78.
- 79.
- 80.
- 81. Digital Authoring Toolsfor AR Vuforia Studio Lens Studio • Support visual authoring of marker- based and/or marker-less AR apps • Provide default markers and support for custom markers • Typically enable AR previews through emulator but need to deploy to AR device for testing
- 82. AR Visual Programming •Rapid prototype on desktop • Deliver on mobile • Simple interactivity • Examples • Zapworks Studio • https://zap.works/studio/ • Snap Lens Studio • https://lensstudio.snapchat.com/ • Facebook Spark AR Studio • https://sparkar.facebook.com/ar-studio/
- 83. Zappar • Zapworks Studio •Code-free interactivity • Desktop authoring for mobile AR • Integrated computer vision (ARkit, ARCore) • Scripting, visual programming • Multiple publishing options • Zappar App, WebAR, App enabled • Zapbox • Inexpensive mobile AR HMD solution • Two handed input ZapBox
- 84.
- 85. Snap LensStudio -https://lensstudio.snapchat.com/ Author and preview AR prototypes ● Tool behind Snapchat Lenses, but also a powerful AR prototyping tool ● Can do face (using front camera) and world lenses (rear camera) ● Simulated previews using webcam Deploy and use advanced AR features ● Can deploy to phone running Snapchat app via Snapcode ● Has advanced AR tracking and segmentation capabilities
- 86. Lens Studio –4.0 https://www.youtube.com/watch?v=efu5pvjYtsw
- 87. Lens Studio Features ●SnapML: train and use ML models ● Face landmarks: track 93 points ● Face expressions: detect winks etc. ● Eye tracking: get eye pos/rotation ● Behaviour: set triggers & actions ● UI: add UI widgets without scripting ● Visual scripting: no code via graphs ● Templates: lots of templates and tutorials
- 88. Example: Museum ARExperience Original art piece by Penelope Umbrico on display at UMMA One of the lenses On-site observation & Snapchat statistics https://medium.com/@amcard/we-used-snapchats-lens-studio-to-build-ar-prototypes-and-this-is-what-we-learned-54e7a98a3d4c
- 89. Immersive Authoring Toolsfor AR • Enable visual authoring of 3D content in AR • Make it possible to edit while previewing AR experience in the environment • Provide basic support for interactive behaviors • Sometimes support export to WebXR Apple Reality Composer Adobe Aero
- 90. Creating On Device •AdobeAero •Create AR on mobile devices •Touch based interaction and authoring •Only iOS support for now •https://www.adobe.com/nz/products/aero.html
- 91.
- 92. Apple Reality Composer •Rapidly create 3D scenes and AR experiences • Creation on device (iPhone, iPad) • Drag and drop interface • Loading 2D/3D content • Simple interactivity – trigger/action • Anchor content in real world (AR view) • Planes (vertical, horizontal), faces, images
- 93.
- 94.
- 95.
- 96.
- 97. XR Tools Landscape Digital& Immersive Authoring Proto.io, Tour Creator, ... Tilt Brush, Blocks, Quill, … Web-Based Development THREE.js, Babylon.js, … A-Frame, AR.js, … Cross-Platform Development Unity + SDKs Unreal + SDKs Native Development Cardboard/Oculus/Vive/... SDK Vuforia/ARCore/ARKit/… SDK
- 98. XR Tools Landscape Digital& Immersive Authoring Good for storyboarding but limited support for interactions Web-Based Development Good for basic XR apps but often interactions feel unfinished Native Development Good for full-fledged XR apps but limited to a particular platform Cross-Platform Development Good for full-fledged XR apps but usually high learning curve
- 99.
- 100. XR Toolkits Card- board AR Kit AR Core Oculus VIVE Holo Lens WMR Web Cam A-Frame AR.js SteamVR MRTK Vuforia ARFoundation XR Interaction WebXR
- 101. XR Toolkits Card- board AR Kit AR Core Oculus VIVE Holo Lens WMR Web Cam A-Frame AR.js SteamVR MRTK Vuforia ARFoundation XR Interaction WebXR
- 102. WebXR: A-Frame • Basedon Three.js and WebGL • New HTML tags for 3D scenes • A-Frame Inspector (not editor) • Asset management (img, video, audio, & 3D models) • ECS architecture with many open- source components • Cross-platform XR
- 103. Hello World inA-Frame https://aframe.io/examples/showcase/helloworld/ HTML code
- 104. XR Toolkits Card- board AR Kit AR Core Oculus VIVE Holo Lens WMR Web Cam A-Frame AR.js SteamVR MRTK Vuforia ARFoundation XR Interaction WebXR
- 105. AR.js – WebXRTracking • Web based AR tracking library • Marker tracking: ARToolkit markers • Image tracking: Natural feature tracking • Location tracking: GPS and compass • Key Features • Very Fast : It runs efficiently even on phones • Web-based : It is a pure web solution, so no installation required. • Full javascript based on three.js + A-Frame + jsartoolkit5 • Open Source : It is completely open source and free of charge! • Standards : It works on any phone with webgl and webrtc • See https://ar-js-org.github.io/AR.js-Docs/
- 107. Source Code <!doctype html> <htmllang=”en”> <head> <script src=”https://aframe.io/releases/0.7.0/aframe.min.js” ></script> <script src=”https://jeromeetienne.github.io/AR.js/aframe/build/aframe-ar.js” ></script> </head> <body style =’margin : 0px; overflow: hidden;’ > <a-scene embedded arjs=’sourceType: webcam;’ > <a-box position=”0 0.5 -0.5″ color=”blue” scale=”1 0.5 1″ rotation= “0 45 0” material=’opacity: 0.9;’ ></a-box> <a-marker-camera preset=’hiro’ ></a-marker-camera> </a-scene> </body> </html> • For more info see https://aframe.io/blog/arjs/
- 108.
- 109. Unity – unity.com •Started out as game engine • Has integrated support for many types of XR apps • Powerful scene editor • Asset management & store • Basically all XR device vendors provide Unity SDKs
- 111. XR Toolkits Card- board AR Kit AR Core Oculus VIVE Holo Lens WMR Web Cam A-Frame AR.js SteamVR MRTK Vuforia ARFoundation XR Interaction WebXR
- 112. Vuforia • Highly optimizedcomputer vision tracking • Multiple types of tracking • Image tracking, object tracking, model tracking, area tracking, etc. • Interaction features • Virtual buttons, occlusion, visual effects, • Multi-platform • Mobile AR, AR headsets See https://www.vuforia.com/
- 114. XR Toolkits Card- board AR Kit AR Core Oculus VIVE Holo Lens WMR Web Cam A-Frame AR.js SteamVR MRTK Vuforia ARFoundation XR Interaction WebXR
- 115. AR Foundation • Aunified Framework for AR • Multi-platform API • Includes core features from ARKit, ARCore, Magic Leap, and HoloLens • Set of behaviours and API with following features • Tracking, light estimation, occlusion, meshing , video pass-through, etc. • Integrates with Unity MARS • See https://unity.com/unity/features/arfoundation
- 117. Unity XR InterationToolKit (preview package) • Easy way to add interactivity to AR/VR experience • Object interactions • UI interactions • Locomotion • Enabling common interactions without writing code • AR gesture, object placement, annotations • https://docs.unity3d.com/Packages/com.unity.xr.interaction.toolkit@1.0/
- 118.
- 119. Unity MARS • Features •Visually author AR apps (WYSIWYG) • Test apps in Unity editor • Develop apps that can interact with real world • Intelligent real-world recognition • Multi-platform development • Based on ARFoundation • ARKit, ARCore, Magic Leap and Hololens • See unity.com/mars
- 120.
- 121. XR Toolkits Card- board AR Kit AR Core Oculus VIVE Holo Lens WMR Web Cam A-Frame AR.js SteamVR MRTK Vuforia ARFoundation XR Interaction WebXR
- 122. Mixed Reality ToolKit(MRTK) • Open-Source Mixed Reality ToolKit • Set of Unity modules/Unreal plugin • Interaction Models • Controllers, gesture, gaze, voice, etc. • UX elements • Foundational elements • Material, text, light, etc. • Controls and behaviours • button, menu, slider, etc. • Tutorials, documentation, guidelines • See https://github.com/microsoft/MixedRealityToolkit-Unity
- 125. Unity vs. A-Frame Unityis a game engine and XR dev platform ●De facto standard for XR apps ●Increasingly built-in support ●Most “XR people” will ask you about your Unity skills :-) Support for all XR devices ●Basically all AR and VR device vendors provide Unity SDKs A-Frame is a declarative WebXR framework ●Emerging XR app development framework on top of THREE.js ●Good for novice XR designers with web dev background Support for most XR devices ●Full WebXR support in Firefox, Chrome, & Oculus Browser
- 126. Conclusions • Prototyping vitalpart of AR Interface Design • Focus on user experience • Rapidly design/test/iterate • Wide variety of Prototyping Tools • Low fidelity • ”Design at the speed of thought” • Sketching, wireframes, mock-ups, layout tools, video prototypes • High fidelity • “Transitions and Final form” • Interactive prototypes, visual authoring, development tools
- 127.
- 128. XR Prototyping WebSite XR Prototyping resources (http://xr-prototyping.org/)
- 129.