22IT009-EXTENDED REALITY

-Vimit Varghese AP/M.Tech CSE

 Introduction of Augmented Reality
 Augmented reality is an interactive
experience that enhances the real world
with computer-generated perceptual
information.
 Using software, apps, and hardware such
as AR glasses, augmented reality overlays
digital content onto real-life environments
and objects.
History of Augmented Reality
 When was augmented reality
invented?
 Augmented reality technology was
invented in 1968, with Ivan Sutherland’s
development of the first head-mounted
display system.
 The technology has come a long way
with a growing list of use cases for
AR.
Augmented reality in the 60s & 70s
 1994: Julie Martin, a writer and producer,
brought augmented reality to the
entertainment industry for the first time
with the theater production titled Dancing
in Cyberspace.
 1999: NASA created a hybrid synthetic
vision system of their X-38 spacecraft.
The system leveraged AR technology to
assist in providing better navigation during
their test flights.
 2000: Hirokazu Kato developed an
open-source software library called the
ARToolKit. This package helps other
developers build augmented reality
software programs. The library uses video
tracking to overlay virtual graphics on top
of the real world.
 2003: Sportvision enhanced the 1st &
Ten graphic to include the feature on
the new Skycam system – providing
viewers with an aerial shot of the field with
graphics overlaid on top of it.
2009: Esquire Magazine
used augmented reality
in print media for the first
time in an attempt to
make the pages come
alive
System Strucure of Augmented Reality
System Structure of Augmented Reality
 Key Technology in AR
 Augmented reality (AR) relies on several
key technologies to overlay digital
information onto the real world.
 These include sensors and cameras for
input, processing units for interpreting
data, and display interfaces to render the
augmented experience.
 Key technologies include SLAM
(Simultaneous Localization and Mapping),
computer vision, computer graphics, and
intelligent interaction technology.
1. Input Technologies (Sensors and Cameras):
Cameras:
Capture the real-world environment, providing visual input
for AR applications.
Sensors:
Include accelerometers, gyroscopes, and GPS, which help
determine the device's position, orientation, and movement
in 3D space.
SLAM (Simultaneous Localization and Mapping):
A core technology that enables AR devices to create a map
of the environment while simultaneously tracking the
device's position within that map.
2. Processing Technologies:
Computer Vision:
Enables the AR system to understand and interpret the
visual data captured by cameras, including object
recognition and scene understanding.
3D Modeling and Rendering:
Used to create the digital objects and environments that are
overlaid onto the real world.
Machine Learning:
Used for tasks like object recognition, gesture recognition,
and improving the accuracy of localization.
3. Display Technologies:
Head-Mounted Displays (HMDs):
Like smart glasses, these provide a direct view of the real
world overlaid with digital content.
Handheld Devices:
Smartphones and tablets can also display AR content,
utilizing their built-in cameras and screens.
Monitors:
Can be used to project AR content onto a surface, creating
a more immersive experience.
4. Interaction Technologies:
Gesture Recognition: Allows users to
interact with AR elements using hand
movements and gestures.
Voice Recognition: Enables users to
control AR applications with voice
commands.
Touch Interaction: Users can interact with
AR content through touch screens or other
touch-sensitive surfaces.
Augmented Reality vs Virtual Reality
 Display Techniques

Monitor Based
Display techniques that are
monitor-based refer to the
various technologies used to
produce visual output on
traditional screens such as
computer monitors, TVs, and
digital signage.
 1. CRT (Cathode Ray Tube)

Oldest monitor technology. Uses electron

beams to excite phosphors on the screen.
Features:
 Bulky and heavy
 Good color reproduction and refresh rates
 Largely obsolete today
Use Case: Legacy systems, retro gaming.
 2. LCD (Liquid Crystal Display)
Description:
Uses liquid crystals sandwiched between layers of glass or plastic, with
a backlight (usually LED).

Features:
Slim and energy-efficient
Lower heat output
Common in modern monitors and TVs

Subtypes:
TN (Twisted Nematic): Fast response, lower color quality (gaming)
IPS (In-Plane Switching): Better color and viewing angles (professional
work)
VA (Vertical Alignment): Deep blacks, good contrast (TVs)
 Head Mounted Displays

Head-Mounted Displays (HMDs) are

wearable display devices worn on the head
or as part of a helmet, primarily used in
Virtual Reality (VR), Augmented Reality
(AR), Mixed Reality (MR), and specialized
applications like military, aviation, and
medical training.
 1. Optical See-Through HMD
• Used in: Augmented Reality (AR)
• Allows user to see the real world directly through a
transparent display.
• Digital content is overlaid on top of the real-world view.

Examples:
Microsoft HoloLens
Magic Leap
Epson Moverio
 Augmented Reality Content

Creating Visual Content-

Creating visual content in Augmented
Reality (AR) involves designing and
integrating digital elements (like 3D models,
text, images, or animations) that overlay or
interact with the real world through AR-
capable devices.
 2. Video See-Through HMD
• Used in: Augmented Reality and Mixed Reality
• Captures real-world view using cameras, then
blends it with virtual content on screens.
• More immersive and flexible but introduces
processing delay.
Examples:
Meta Quest Pro (in MR mode)
Varjo XR-4
 Issues and Challenges in AR
 Augmented Reality (AR) faces several challenges
including hardware limitations, software development
complexities, content creation, user experience issues,
and ethical/legal concerns.
 Specifically, AR hardware struggles with processing
power, battery life, and ergonomics, while software
development must overcome integration challenges and
ensure seamless performance. Content creation requires
significant expertise and resources, and user experience
can be hindered by factors like limited field of view, low
resolution, and potential for motion sickness.
 Finally, data privacy, public acceptance, and potential for
physical harm are key ethical and safety considerations.
 Augmented Reality Content

Creating Visual Content

Creating visual content in Augmented
Reality (AR) involves designing and
integrating digital elements (like 3D models,
text, images, or animations) that overlay or
interact with the real world through AR-
capable devices.
1.Define Your Objective and Plan:
• What problem are you solving or what
experience are you creating?
• Determine the purpose of your AR content and
the specific outcome you want to achieve.
• Who is your target audience? Consider their
technical capabilities and what kind of
experience will resonate with them.
• What are your key performance indicators
(KPIs)? How will you measure the success of
your AR experience?
2.Design the AR Content and User Interface:
Create 3D models and animations:
Use tools like Blender, Maya, or specialized AR creation
software to build your 3D assets. Consider using asset
stores for ready-made 3D models and animations.
Design the user interface (UI) and user experience
(UX):
Ensure the UI is intuitive and easy to navigate, and the UX
is engaging and immersive.
Consider visual design principles:
Focus on readability, scale, and how the AR content
interacts with the real world.
3. Choose Technologies and Development
Approach:
• Select an AR development platform: Consider
using ARKit (for iOS), ARCore (for Android), or
platforms like Unity or Vuforia.
• Decide on the development approach: Will
you build a native AR application, a web-based
AR experience (WebAR), or use a platform like
Snapchat Lenses?
• Consider the target devices: Will your
experience be used on smartphones, tablets, or
AR glasses?
4.Develop and Test:
Build the AR experience:
Follow the chosen development approach and utilize the
selected technologies to create your AR content.
Test on different devices:
Ensure your AR content works correctly on various devices
and operating systems.
Gather user feedback:
Test your experience with target users and iterate based on
their feedback.
Optimize performance:
Ensure your AR experience is efficient and doesn't drain
battery life or cause performance issues.
5. Release and Promote:
Release your AR experience:
Publish your app to app stores, deploy your
WebAR experience, or create Snapchat
lenses.
Promote your AR content:
Use social media, your website, and other
digital channels to reach your target
audience.
 Creating Audio Content
 Augmented reality (AR) audio creates
immersive experiences by layering virtual
sounds onto the real world, enhancing
user engagement and providing new ways
to interact with digital content.
 This technology, also known as Audio
Augmented Reality (AAR), uses spatial
audio to make virtual sounds appear to
originate from specific locations in the
user's environment, creating a more
realistic and engaging experience.
1. Enabling Technologies:
• Smart headphones:
These headphones, often with built-in sensors and spatial audio
capabilities, are crucial for delivering the virtual sounds to the user.
• Voice recognition and synthesis:
These technologies allow for interactive experiences where users can
speak commands or receive spoken information within the AR
environment.
• Large language models (LLMs):
LLMs can be used to generate dynamic and context-aware audio
content, making interactions more natural and personalized.
• Contextual data:
Information about the user's location, environment, and interactions is
used to tailor the audio experience, ensuring it aligns with the user's
current context.
2. Creating Spatial Audio:
• 3D audio engines:
These engines are used to create the illusion of sounds coming from
specific locations in 3D space. This is achieved through techniques like
panning, which adjusts the volume and direction of sounds based on
the user's position.
• Mono audio files:
For best results, mono audio files are often preferred as they allow the
3D audio engine to create more realistic panning effects.
• Audio layering:
Different audio clips can be layered together to create more complex
and immersive soundscapes. This can include ambient sounds, object-
specific sounds, and interactive elements.
 Creating content for Other Senses(Touch, Taste,
Smell)
• Augmented reality (AR) is primarily known for
augmenting the visual and auditory senses, but
it's actively exploring the potential to enhance
touch, taste, and smell as well.
• While visual and auditory enhancements are
more common in current AR applications,
researchers are developing technologies to
incorporate haptic (touch), olfactory (smell), and
gustatory (taste) experiences.
1. Touch (Haptic):
• AR can use devices like gloves or suits
equipped with actuators to simulate
physical sensations.
• These devices can create the feeling of
pressure, texture, or even temperature
changes when interacting with virtual
objects.
• For example, users could feel the texture
of a virtual fabric or the resistance of a
virtual object when pushing it.
2. Smell (Olfactory):
• AR can use devices that release specific
scents to create an olfactory experience.
• These devices can be integrated into
headsets or other wearable technology.
• Imagine smelling freshly baked bread
while viewing a virtual bakery or
experiencing the scent of a forest during a
virtual hike.
3. Taste (Gustatory):
• AR's ability to augment taste is still in its
early stages, but research is exploring
ways to stimulate taste receptors.
• This could involve using electrical or
chemical stimulation to trick the taste
buds.
• For instance, users might "taste" different
flavors while drinking plain water, or
experience a virtual meal with various
flavor profiles.
 Mobile Augmented Reality
• Mobile AR, or mobile augmented reality, refers to
the use of augmented reality (AR) technology on
smartphones and tablets, allowing users to
overlay digital content onto the real world they
see through their device's camera.
• This technology transforms the mobile device
into a portal for interacting with virtual elements
within the physical environment, enhancing user
experiences in various applications.
 Working
1.Camera input:
The device's camera captures the real-world environment.
2.Image recognition and tracking:
Algorithms analyze the camera feed to identify and track
objects, surfaces, and locations in the real world.
3.Rendering and display:
Digital content is rendered and displayed on the device's
screen, overlaid onto the camera's view.
4.User interaction:
Touch input, gestures, and voice commands allow users to
interact with the AR elements.
 Examples of mobile AR applications:

Gaming:
Pokémon Go overlays virtual creatures onto the real world, allowing
players to capture them in their environment.
Retail:
Apps like Wayfair allow users to virtually "place" furniture in their
homes before buying it.
Education:
AR can make complex concepts more tangible, like visualizing the
human anatomy or historical events.
Navigation:
Google Maps' Live View uses AR to overlay navigation instructions
onto the real-world view.
Entertainment:
AR apps can create interactive experiences in museums, parks, and
other public spaces.