Download as PDF, PPTX
Uploaded byMark Billinghurst
COMP 4010 - Lecture 4: 3D User Interfaces
The document discusses 3D user interfaces in virtual reality, highlighting the importance of natural interaction and immersion. It outlines various interaction techniques, including selection and manipulation methods, as well as travel and wayfinding strategies within virtual environments. The document emphasizes the need for effective design guidelines to improve user experience and usability in VR applications.
More Related Content
Similar to COMP 4010 - Lecture 4: 3D User Interfaces
COMP 4010 - Lecture 4: 3D User Interfaces
- 1. LECTURE 4: 3DUSER INTERFACES FOR VR COMP 4010 – Virtual Reality Semester 5 – 2018 Bruce Thomas, Mark Billinghurst, Gun Lee University of South Australia August 14th 2018
- 2. VR Paper Presentations– August 14th • Ian Hanan • IEEE transactions on visualization and computer graphics. 2018 Apr 1;24(4) Collision avoidance behavior between walkers: global and local motion cues • Christian Bonato • IEEE transactions on visualization and computer graphics. 2018 Apr 1;24(4) Exercise Intensity-driven Level Design
- 3. Lecture 3: Recap- Tracking • Registration vs. Tracking • Tracking Devices • Mechanical • Magnetic • Inertial • Optical • Inside Out vs. Outside In • Windows Mixed Reality • HTC Vive Lighthouse
- 4. Lecture 3: Recap •VR Input Devices • Body, Feet, Handheld, Gestures, Gaze • Separate device from Interaction Technique • VR Systems • Multiple components • Content, Software, User I/O modules • VR Simulation Loop • System delays cause sickness • Reduce system delay • Predictive tracking, faster components
- 5.
- 6. Typical Virtual RealitySystem HMD Input Tracking User Interface
- 7. How can weInteract in VR? • How can VR devices create a natural user experience?
- 8. Background: Human-computer interaction •HCI studies communication • Users and computers communicate via the interface • Traditional UI design issues: • Input device • Interaction style • Feedback to the user • Gulf of execution / gulf of evaluation • All these are relevant for 3D/VR User Interfaces
- 9. Why 3D Interaction? •3D / VR application should be useful • Support immersion • Use natural skills • Provide immediacy of visualization • But many current VR apps either • Support only simple interaction • Or, have serious usability problems • We need good 3D user interface guidelines
- 10. Some Definitions • 3DInteraction: • Human-computer interaction in which the user’s tasks are carried out in a 3D spatial context • 3D input devices, 2D input devices mapping into 3D • 3D user interface (3D UI): • A UI that involves 3D interaction • 3D interaction technique: • A method (hardware and software) allowing a user to accomplish a task in a 3D UI 10
- 11. Examples of 3DUIs – VR and non-VR 3D physical input, 3D virtual context 3D physical input, 2D virtual context 2D physical input, 3D virtual context
- 12. What makes 3Dinteraction difficult? • Spatial input • Lack of constraints • Lack of standards • Lack of tools • Lack of precision • Fatigue • Layout more complex • Perception
- 13.
- 14. Moving from Menusto Natural Interaction 2D UI in 3D Natural 3D Interaction
- 15. Natural Interface Concept- WorldBuilder • https://www.youtube.com/watch?v=FheQe8rflWQ&t=43s
- 16. World Builder Today(Available on Steam) • https://www.youtube.com/watch?v=65u3W7wjXs0
- 17. Vision vs. Reality– Still Work to Do.. Natural interface Gesture, speech Wide field of view Full body input Limited input Wireless, limited range tracking Reduced field of view 2D GUI in VR
- 18. Universal 3D InteractionTasks in VR • Object Interaction • Selection: Picking object(s) from a set • Manipulation: Modifying object properties • Navigation • Travel: motor component of viewpoint motion • Wayfinding: cognitive component; decision-making • System control • Issuing a command to change system state or mode
- 19.
- 20. Selection and Manipulation •Selection: • specifying one or more objects from a set • Manipulation: • modifying object properties • position, orientation, scale, shape, color, texture, behavior..
- 21. Goals of selection •Indicateaction on object •Query object •Make object active •Travel to object location •Set up manipulation
- 22. Selection performance • Variablesaffecting user performance • Object distance from user • Object (visual) size • Density of objects in area • Occluders
- 23. Classification of SelectionTechniques • asdf
- 24. 24 Selection classification Selection Feedback Object indication Indication to select graphical tactile audio objecttouching pointing indirect selection button gesture voice
- 25. Common Selection Techniques •Simplevirtual hand •Ray-casting •Occlusion •Go-go (arm-extension)
- 26. Simple virtual handtechnique • Process • One-to-one mapping between physical and virtual hands • Object can be selected by “touching” with virtual hand • “Natural” mapping • Limitation: • Only select objects in hand reach
- 27. Ray-casting technique • “Laserpointer” attached to virtual hand • First object intersected by ray may be selected • User only needs to control 2 DOFs • Proven to perform well for remote selection • Variants: • Cone casting • Snap-to-object rays
- 28. Example Ray Casting •https://www.youtube.com/watch?v=W1ZUBTPCL3E
- 29. Occlusion technique • Image-planetechnique - truly 2D • Occlude/cover desired object with selector object (e.g. finger) • Nearest object along ray from eye through finger may be selected
- 30. Image Plane Interaction •Pierce, J., Forsberg, A., Conway, M., Hong, S., Zeleznik, R., & Mine, M. (1997). Image Plane Interaction Techniques in 3D Immersive Environments. Proceedings of the ACM Symposium on Interactive 3D Graphics, 39-44.
- 31.
- 32. Go-Go Technique • Arm-extensiontechnique • Non-linear mapping between physical and virtual hand position • Local and distant regions (linear < D, non-linear > D) Poupyrev, I., Billinghurst, M., Weghorst, S., & Ichikawa, T. (1996). The Go-Go Interaction Technique: Non-linear Mapping for Direct Manipulation in VR. Proceedings of the ACM Symposium on User Interface Software and Technology, 79-80.
- 33. Precise 3D selectiontechniques • Increase selection area • Cone-casting (Liang, 1993) • Snapping (de Haan, 2005) • 3D Bubble Cursor (Vanacken, 2007) • Sphere-casting (Kopper 2011) • Increase control/display ratio • PRISM (Frees, 2007) • ARM (Kopper, 2010) Not ideal for cluttered environments (high density, occlusion) May require careful interaction
- 34.
- 35. Sphere-casting (SQUAD) • Twophases • Sphere-casting followed by QUAD-menu selection • Features • Multiple low precision selections • Scales well – at most log4n+1 refinement steps • Limitations • Quad-menu phase is done outside spatial context • Target needs to be unique or selectable among identical ones 35 Kopper, R., Bacim, F., & Bowman, D. A. (2011). Rapid and accurate 3D selection by progressive refinement. In 3D User Interfaces (3DUI), 2011 IEEE Symposium on (pp. 67-74). IEEE.
- 36. SQUAD Selection Sphere casting QuadSelection
- 37.
- 38. PRISM (Frees 2005) •Change Control/Gain ratio based on hand speed • As hand moves slower, scale down object motion • As hand moves faster, us 1:1 motion mapping • Twice the performance for object docking tasks Frees, S., & Kessler, G. D. (2005). Precise and rapid interaction through scaled manipulation in immersive virtual environments. In Virtual Reality, 2005. Proceedings. VR 2005. IEEE (pp. 99-106).
- 39. Goals of manipulation •Objectplacement • Design • Layout • Grouping •Tool usage •Travel
- 40. Classification of ManipulationTechniques • asdfa
- 41. Technique Classification byComponents Manipulation Object Attachment Object Position Object Orientation Feedback attach to hand attach to gaze hand moves to object object moves to hand user/object scaling no control 1-to-N hand to object motion maintain body-hand relation other hand mappings indirect control no control 1-to-N hand to object rotation other hand mappings indirect control graphical force/tactile audio
- 42. Common Manipulation Techniques •Simplevirtual hand •HOMER •Scaled-world grab •World-in-miniature
- 43. Simple Virtual HandManipulation https://www.youtube.com/watch?v=_OgfREa4ggw
- 44. HOMER technique Hand-Centered Object Manipulation Extending Ray-Casting • Selection:ray-casting • Manipulate: directly with virtual hand • Include linear mapping to allow wider range of placement in depth Time Bowman, D., & Hodges, L. (1997). An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments. Proceedings of the ACM Symposium on Interactive 3D Graphics, 35-38.
- 45.
- 46. Scaled-world Grab Technique •Often used w/ occlusion • At selection, scale user up (or world down) so that virtual hand is actually touching selected object • User doesn‘t notice a change in the image until he moves Mine, M., Brooks, F., & Sequin, C. (1997). Moving Objects in Space: Exploiting Proprioception in Virtual Environment Interaction. Proceedings of ACM SIGGRAPH, 19-26
- 47. World-in-miniature (WIM) technique •“Dollhouse” world held in user’s hand • Miniature objects can be manipulated directly • Moving miniature objects affects full-scale objects • Can also be used for navigation Stoakley, R., Conway, M., & Pausch, R. (1995). Virtual Reality on a WIM: Interactive Worlds in Miniature. Proceedings of CHI: Human Factors in Computing Systems, 265-272, and Pausch, R., Burnette, T., Brockway, D., & Weiblen, M. (1995). Navigation and Locomotion in Virtual Worlds via Flight into Hand-Held Miniatures. Proceedings of ACM SIGGRAPH, 399-400.
- 48.
- 49. Voodoo Doll Interaction •Manipulate miniature objects • Act on copy of objects • Actions duplicated on actual object • Supports action at a distance • Two handed technique • One hand sets stationary reference frame • Second hand manipulates object Pierce, J. S., Stearns, B. C., & Pausch, R. (1999). Voodoo dolls: seamless interaction at multiple scales in virtual environments. In Proceedings of the 1999 symposium on Interactive 3D graphics (pp. 141-145). ACM.
- 50. 50 Two-Handed Interaction • Symmetricvs. Asymmetric • Symmetric: both hands performing same actions • Asymmetric: both hands performing different actions • Dominant (D) vs. non-dominant (ND) hand • Guiard’s principles • ND hand provides frame of reference • ND hand used for coarse tasks, D hand for fine-grained tasks • Manipulation initiated by ND hand Guiard, Y., "Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model," J. Motor Behavior, 19 (4), 1987, pp. 486-517.
- 51. Symmetric Bimanual Technique •iSith (Wyss 2006) • Using two 6 DOF controllers each ray casting • Intersection point of two rays determines interaction point Wyss, H. P., Blach, R., & Bues, M. (2006, March). iSith-Intersection-based spatial interaction for two hands. In 3D User Interfaces, 2006. 3DUI 2006. IEEE Symposium on (pp. 59-61). IEEE.
- 52. Asymmetric Bimanual Technique •Spindle + Wheel (Cho 2015) • Two 6 DOF handheld controls • One dominant, one ND • Movement one hand relative to other provides 7 DOF input Cho, I., & Wartell, Z. (2015). Evaluation of a bimanual simultaneous 7DOF interaction technique in virtual environments. In 3D User Interfaces, 2015 IEEE Symposium on (pp. 133-136). IEEE.
- 53. Demo: Spindle +Wheel 7 DOF Input • https://www.youtube.com/watch?v=nF4egFHyLYM
- 54. Design Guidelines forManipulation • There is no single best manipulation technique • Map the interaction technique to the device • Reduce degrees of freedom when possible • Use techniques that can help to reduce clutching • Consider the use of grasp-sensitive object selection • Use pointing techniques for selection and grasping techniques for manipulation • Use existing techniques unless there is a large amount of benefit from designing a new application-specific method • Consider the trade-off between technique design and environmental design
- 55.
- 56. Navigation • How wemove from place to place within an environment • The combination of travel with wayfinding • Wayfinding: cognitive component of navigation • Travel: motor component of navigation • Travel without wayfinding: "exploring", "wandering”
- 57. Travel • The motorcomponent of navigation • Movement between 2 locations, setting the position (and orientation) of the user’s viewpoint • The most basic and common VE interaction technique, used in almost any large-scale VE
- 58. Types of Travel •Exploration • No explicit goal for the movement • Search • Moving to specific target location • Naïve – target position not known • Primed – position of target known • Maneuvering • Short, precise movements changing viewpoint
- 59. Movement Process • Focusingon user control
- 60. Technique classification • Physicallocomotion metaphors • treadmills, cycles, etc… • Steering metaphor • Route planning metaphor • Target specification metaphor • Manual manipulation metaphor • Scaling metaphor 60
- 61.
- 62. Classification of Traveland Locomotion Virtual turning Real turning Virtual translation Desktop VEs Vehicle simulators CAVE wand Most HMD systems Walking in place Magic Carpet Real translation Stationary cycles Treadport Biport Wide-area tracking UNIPORT ODT Can classify locomotion devices in terms of real vs. virtual travel
- 63. Taxonomy of TravelTechniques • Focusing on sub-task of travel Bowman, D. A., Koller, D., & Hodges, L. F. (1997, March). Travel in immersive virtual environments: An evaluation of viewpoint motion control techniques. In Virtual Reality Annual International Symposium, 1997., IEEE 1997 (pp. 45-52). IEEE.
- 64. Gaze Directed Steering •Move in direction that you are looking • Very intuitive, natural navigation • Can be used on simple HMDs (e.g. Google Cardboard) • But: Can’t look in different direction while moving
- 65. Example: Gaze DirectedSteering • https://www.youtube.com/watch?v=6iKxser1Wic
- 66. TelePortation • Use controllerto select end point • Usable with 3DOF contoller • Jump to a fixed point in VR • Discrete motion can be confusing/cause sickness
- 67.
- 68. Pointing Technique • A“steering” technique • Use hand tracker instead of head tracker • Point in direction you want to go • Slightly more complex, than gaze-directed steering • Allows travel and gaze in different directions • good for relative motion, look one way, move another
- 69. Example: VIVE Thrust •https://www.youtube.com/watch?v=JRgCe_8q4vE
- 70. Grabbing the AirTechnique • Use hand gestures to move yourself through the world • Metaphor of pulling a rope • Often a two-handed technique • May be implemented using Pinch Gloves Mapes, D., & Moshell, J. (1995). A Two-Handed Interface for Object Manipulation in Virtual Environments. Presence: Teleoperators and Virtual Environments, 4(4), 403-416.
- 71. Moving Your OwnBody • Can move your own body • In World in Miniature, or map view • Grab avatar and move to desired point • Immediate teleportation to new position in VE Moving avatar in Map View Moving avatar in WIM view
- 72. Example: Navigation UsingWIM • https://www.youtube.com/watch?v=VxGqIjMlTs8
- 73. Walking in Place •https://www.youtube.com/watch?v=J_yQfW1qYGI
- 74. Redirected Walking • Addressproblem of limited walking space • Warp VR graphics view of space • Create illusion of walking straight, while walking in circles Razzaque, S., Kohn, Z., & Whitton, M. C. (2001, September). Redirected walking. In Proceedings of EUROGRAPHICS (Vol. 9, pp. 105-106).
- 75.
- 76. Redirected Walking withWalls • https://www.youtube.com/watch?v=u8pw81VbMUU
- 77. Guided Navigation Technique •Water skiing metaphor for VR movement • Good for moving in a fixed direction, while giving user some control
- 78.
- 79. Wayfinding • The meansof • determining (and maintaining) awareness of where one is located (in space and time), • and ascertaining a path through the environment to the desired destination • Problem: 6DOF makes wayfinding hard • human beings have different abilities to orient themselves in an environment, extra freedom can disorient people easily • Purposes of wayfinding tasks in virtual environments • Transferring spatial knowledge to the real world • Navigation through complex environments in support of other tasks
- 80. Wayfinding – MakingCognitive Maps • Goal of Wayfinding is to build Mental Model (Cognitive Map) • Types of spatial knowledge in a mental model • landmark knowledge • procedural knowledge (sequence of actions required to follow a path) • map-like (topological) knowledge • Creating a mental model • systematic study of a map • exploration of the real space • exploration of a copy of the real space • Problem: Sometimes perceptual judgments are incorrect within a virtual environment • e.g. users wearing a HMD often underestimate dimensions of space, possibly caused by limited field of view
- 81. Kevin Lynch –The Image of the City • In real cities, five elements • Path, Edge, District, Node, Landmark • VR environments the same
- 83. Designing VE toSupport Wayfinding • Provide Landmarks • Any obvious, distinct and non-mobile object can serve as a landmark • A good landmark can be seen from several locations (e.g. tall) • Audio beacons can also serve as landmarks • Use Maps • Copy real world maps • Ego-centric vs. Exocentric map cues • World in Miniature • Map based navigation
- 84. Wayfinding Aids • Pathfollowing • Easy method of wayfinding • Multiple paths through a single space may be denoted by colors • For example, hospitals that use colored lines to indicate how to get to certain locations. • Bread crumbs (leaving a trail) • leaving a trail of markers - like Hänsel and Gretel • allows participant to know when they've been somewhere before • having too many markers can make the space be overly cluttered • Compass • may also be other form of direction indicator (e.g. artificial horizon) • may specify directions in 2D space or 3D space
- 85.
- 86. Design Guidelines forNavigation • Match the travel technique to the application • Use an appropriate combination of travel technique, display devices, and input devices • The most common travel tasks should require a minimum of effort from the user • Use physical locomotion technique if user exertion or naturalism is required • Use target-based techniques for goal-oriented travel and steering techniques for exploration and search • Provide multiple travel techniques to support different travel tasks in the same application • Choose travel techniques that can be easily integrated with other interaction techniques in the application
- 87.
- 88. System Control • Issuinga command to change system state or mode • Examples • Launching application • Changing system settings • Opening a file • Etc. • Key points • Make commands visible to user • Support easy selection
- 89.
- 90. Example: GearVR Interface •2D Interface in 3D Environment • Head pointing and click to select
- 91.
- 92. TULIP Menu • Menuitems attached to virtual finger tips • Ideal for pinch glove interaction • Use one finger to select menu option from another Bowman, D. A., & Wingrave, C. A. (2001, March). Design and evaluation of menu systems for immersive virtual environments. In Virtual Reality, 2001. Proceedings. IEEE (pp. 149-156). IEEE.
- 93. 2D Menus inVR • Many examples of 2D GUI and floating menus in VR Nested Pie Menu 2D Menu in VR CAVE
- 94. Example: Marking Menuin VR • https://www.youtube.com/watch?v=BTTBgZ94lAc
- 95. Tools • Use toolsfor system commands • Tangible user interfaces (real tools) • Virtual tools (3D objects) • Design issues • Support eyes-off use • Use of physical affordances • Base on familiar objects • Provide tactile feedback • Map real tool to virtual operation Tangible interface for CAVE
- 96. Voice Input • Implementation •Wide range of speech recognition engines available • E.g. Unity speech recognition plug-in, IBM VR speech sandbox • Factors to consider • Recognition rate, background noise, speaker dependent/independent • Design Issues • Voice interface invisible to user • no UI affordances, overview of functions available • Need to disambiguate system commands from user conversation • Use push to talk or keywords • Limited commands – use speech recognition • Complex application – use conversational/dialogue system
- 97. Example – IBMVR Speech Sandbox • https://www.youtube.com/watch?v=NoO2R3Pz5Go • Available from: http://ibm.biz/vr-speech-sandbox
- 98. Design Guidelines forSystem Control • Avoid mode errors • Design for discoverability • Consider using multimodal input • Use an appropriate spatial reference frame • Prevent unnecessary focus and context switching • Avoid disturbing the flow of action of an interaction task • Structure the functions in an application and guide the user • 3D is not always the best solution – consider hybrid interfaces
- 99.
- 100. 100 Conclusions lUsability one ofthe most crucial issues facing VE applications lImplementation details critical to ensure usability lEase of coding not equal to ease of use lSimply adapting 2D interfaces is not sufficient
- 101. Conclusions • User interfacekey for good VR experience • Need 3D user interface techniques • Design for • Selection/Manipulation • Navigation • System control • Follow good design guidelines • Cannot just implement 2D techniques in VR
- 102. Resources • Excellent book •3D User Interfaces: Theory and Practice • Doug Bowman, Ernst Kruijff, Joseph, LaViola, Ivan Poupyrev • Great Website • http://www.uxofvr.com/ • 3D UI research at Virginia Tech. • research.cs.vt.edu/3di/
- 103. UX of VRWebsite - www.uxofvr.com • Many examples of great interaction techniques • Videos, books, articles, slides, code, etc..
- 104. Acknowledgments – ContentFrom • Doug Bowman, Virginia Tech • Joe LaViola, University of Central Florida • Ernst Kruijff, Graz Univ. of Technology • Ivan Poupyrev, Google Doug Bowman
- 105.