AUGMENTED REALITY in UNITY 3D

VIRTUAL AND AUGMENTED REALITY - DIMEG

Prof. F. Bruno
 What is Augmented Reality?
Augmented Reality enhances real world by computer-
generated, multisensory, perceptual information.

VAR – Prof. F. Bruno 2 UNICAL – INGEGNERIA MECCANICA 2020/2021

 How Augmented reality works
• AR systems must derive from real-
world coordinates and camera
images.
• Video tracking and visual
odometry allow the software to
recognize features captured by
the camera, and translate them in
spatial information;
• Once based exclusively on
markers, Augmented reality today
can rely on real world-features
detection.

VAR – Prof. F. Bruno 3 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Toolkits
ARKit, ARCore, and AR Foundation are among the most popular
toolkits that facilitate the tools needed to build an augmented
reality app. These have simplified AR app development and they are
compatible with Unity 3D

ARCore ARKit AR FOUNDATION

VAR – Prof. F. Bruno 4 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Foundation
AR Foundation allows you to work with augmented reality platforms in a
multi-platform way within Unity. This package presents an interface for
Unity developers to use, but doesn't implement any AR features itself.

To use AR Foundation on a target device, you also need separate packages

for the target platforms officially supported by Unity (ARCore, ARKit, Magic
Leap and Windows XR plugins)

https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.1/manual/
index.html

VAR – Prof. F. Bruno 5 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Foundation in Unity (1)
AR Foundation is a set of MonoBehaviours and APIs for dealing with devices that support
the following concepts:
• Device tracking: track the device's position and orientation in physical space.
• Plane detection: detect horizontal and vertical surfaces.
• Point clouds, also known as feature points.
• Anchor: an arbitrary position and orientation that the device tracks.
• Light estimation: estimates for average color temperature and brightness in physical
space.
• Environment probe: a means for generating a cube map to represent a particular area of
the physical environment.
• Face tracking: detect and track human faces.
• 2D image tracking: detect and track 2D images.
• 3D object tracking: detect 3D objects.

VAR – Prof. F. Bruno 6 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Foundation in Unity (2)
• Meshing: generate triangle meshes that correspond to the physical space.
• Body tracking: 2D and 3D representations of humans recognized in physical space.
• Collaborative participants: track the position and orientation of other devices in a shared
AR experience.
• Human segmentation and occlusion: apply distance to objects in the physical world to
rendered 3D content, which achieves a realistic blending of physical and virtual objects.
• Raycast: queries physical surroundings for detected planes and feature points.
• Pass-through video: optimized rendering of mobile camera image onto touch screen as
the background for AR content.
• Session management: manipulation of the platform-level configuration automatically
when AR Features are enable or disabled.
• Occlusion

VAR – Prof. F. Bruno 7 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Feature Support Per Platform
AR Foundation does not implement
any AR features itself but, instead,
defines a multi-platform API that
allows you to work with functionality
common to multiple platforms.

You can refer to this table to

understand which parts of AR
Foundation are relevant on specific
platforms:

VAR – Prof. F. Bruno 8 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Supported Platform Packages
The following platform packages and later implement the AR Foundation
features indicated above:

VAR – Prof. F. Bruno 9 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Requirements
The current version of AR Foundation is compatible with the
following versions of the Unity Editor:
• 2019.4
• 2020.1
• 2020.2
• 2021.1

VAR – Prof. F. Bruno 10 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Subsystems
AR Foundation is built on subsystems. A subsystem is a platform-agnostic
interface for surfacing different types of information. The AR-related
subsystems are defined in the AR Subsystems package and use the
namespace UnityEngine.XR.ARSubsystems. You will occasionally need to
interact with the types in the AR Subsystems package.

Each subsystem handles specific functionality:

For example, XRPlaneSubsystem provides the plane detection interface.

VAR – Prof. F. Bruno 11 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Providers
A provider is a concrete implementation of a subsystem. For example, the ARCore
XR Plugin package contains the ARCore implementation for many of the AR
subsystems.

Because different providers have varying support for specific features, each
subsystem also has a descriptor that indicates which specific subsystem features it
supports. For example, the XRPlaneSubsystemDescriptor contains properties
indicating whether it supports horizontal or vertical plane detection.

Each individual provider determines how to implement each subsystem. In general,

they wrap that platform's native SDK (for example, ARKit on iOS and ARCore on
Android).

VAR – Prof. F. Bruno 12 UNICAL – INGEGNERIA MECCANICA 2020/2021

 How to install AR Foundation
To install this package: open the Package Manager and search AR Foundation. Then click
Install.

VAR – Prof. F. Bruno 13 UNICAL – INGEGNERIA MECCANICA 2020/2021

 How to use AR Foundation
Provider plugins must be enabled before AR Foundation can use them. XR
Plugin Management (Edit->Project Settings) provides a UI to enable specific
plug-in providers for each target platform.

VAR – Prof. F. Bruno 14 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Configuring your Unity Project for XR
To configure your Unity Project for XR, follow these steps:
1. Open the Project Settings window (Edit->Project Settings), and select XR Plug-in
Management.
2. Click Install XR Plug-in Management if the package hasn’t been installed already. You
can also install it from the Package Manager window.
3. After installation completes, select a Plug-in Provider to enable it for the corresponding
build target. To do this:
1. Select a build target (for example, Android).
2. Select the checkbox to the left of each plug-in you want to use for that build target (for example,
ARCore).
4. After a plug-in loads, it displays in the left-hand navigation, under XR Plug-in
Management. Click the plug-in to configure its settings for each build target.

VAR – Prof. F. Bruno 15 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Documentation

https://docs.unity3d.com/Packages/com.unity.xr.arfound
ation@4.2/manual/index.html

VAR – Prof. F. Bruno 16 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Glossary

VAR – Prof. F. Bruno 17 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Scene setup
In order to create a standard AR scene, right-click in the scene hierarchy, and
select XR > AR Session or XR > AR Session Origin from the context menu.

VAR – Prof. F. Bruno 18 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Session
An AR scene should include an ARSession component. The AR Session
controls the lifecycle of an AR experience by enabling or disabling AR on the
target platform. The ARSession can be on any GameObject.
When you disable the ARSession, the system no longer tracks features in its
environment, but if you enable it at a later time, the system will attempt to
recover and maintain previously-detected features.
If you enable the Attempt Update option, the device tries to install AR
software if possible. Support for this feature is platform-dependent

VAR – Prof. F. Bruno 19 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Session Origin
The purpose of the ARSessionOrigin is to transform trackable features, such
as planar surfaces and feature points, into their final position, orientation,
and scale in the Unity Scene. Because AR devices provide their data in
"session space", which is an unscaled space relative to the beginning of the
AR session, the ARSessionOrigin performs the appropriate transformation
into Unity space.
ARSessionOrigin also allows you to scale virtual content and apply an offset
to the AR Camera. If you're scaling or offsetting the ARSessionOrigin, then its
AR Camera should be a child of the ARSessionOrigin. Because the AR Camera
is session-driven, this setup allows the AR Camera and detected trackables to
move together.

VAR – Prof. F. Bruno 20 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Pose Driver / TrackedPoseDriver
The AR Pose Driver drives the local position and orientation of the parent
GameObject according to the device's tracking information. The most
common use-case for this would be attaching the ARPoseDriver to the AR
Camera to drive the camera's position and orientation in an AR scene.

VAR – Prof. F. Bruno 21 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Camera Manager
The ARCameraManager enables
features for the AR Camera,
including the management of the
device camera textures and the
properties that set the light
estimation modes.

VAR – Prof. F. Bruno 22 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Camera Background
If you want the video feed from the device camera to show up as the
rendered background of the scene at runtime, you need to add an
ARCameraBackground component to a Camera. Otherwise, the background
at runtime will come from the Camera.clearFlags setting.
The ARCameraBackground component subscribes to AR Camera events and
renders the AR Camera texture to the screen (that is, the background texture
from the device camera must be rendered for each frame). This is not
required, but common for AR apps.

VAR – Prof. F. Bruno 23 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Input Manager
This component is required to enable world tracking. Without it, the Tracked
Pose Driver can't acquire a pose for the device.
This component can be anywhere in your Scene, but you shouldn't have
more than one.

VAR – Prof. F. Bruno 24 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Trackables Manager
In AR Foundation, a "trackable" is anything that can be detected and tracked
in the real world. Planes, point clouds, anchors, environment probes, faces,
images, and 3D objects are all examples of trackables.

Each trackable has a trackable manager. All the trackable managers must be
on the same GameObject as the AR Session Origin. This is because the
session origin defines the transform to which all the detected trackables are
relative. The trackable managers use the session origin to place the detected
trackables in the correct place in the Unity scene graph.

VAR – Prof. F. Bruno 25 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Trackables Manager

VAR – Prof. F. Bruno 26 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Enabling and disabling features
Enabling a trackable manager enables that feature. For
example, you can toggle plane detection by enabling or
disabling the AR Plane Manager.
Enabling a particular feature can cause the device to
consume more power, so it's best to disable managers
when your application isn't using them.

VAR – Prof. F. Bruno 27 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Trackable lifetime
Each trackable can be added, updated, and removed. Each frame, the managers query for
the set of changes to their trackables since the previous frame. Each manager has an event
to which you can subscribe to be notified of these changes:

VAR – Prof. F. Bruno 28 UNICAL – INGEGNERIA MECCANICA 2020/2021

 AR Foundation examples:
https://github.com/Unity-Technologies/arfoundation-
samples/tree/4.1

VAR – Prof. F. Bruno 29 UNICAL – INGEGNERIA MECCANICA 2020/2021

 Example
At runtime, ARFoundation will generate an
ARTrackedImage for each detected reference image.

This sample uses the TrackedImageInfoManager.cs

script to overlay the original image on top of the
detected image, along with some meta data.

VAR – Prof. F. Bruno 30 UNICAL – INGEGNERIA MECCANICA 2020/2021