PALAS, ANGEL MAE 02/24/2025

IE-PC 3215
Assignment 1

Title: The Influence of Agent Transparency and Complexity on Situation

Awareness, Mental Workload, and Task Performance
Author(s): Koen van de Merwe, Steven Mallam, Salman Nazir, and Oystein
Engelhardtsen
Year: 2024
Journal: Journal of Cognitive Engineering and Decision-Making
Issue No: 2
Volume No: 18

Theoretical foundation:

This study explores agent transparency's role in enhancing situation

awareness for human operators monitoring autonomous ships. The following
theories and models used in this study are:

 Situational Awareness (SA) - SA regarding the system's

solution and information provided was examined using the
Situation Awareness Global Assessment Technique (SAGAT;
Endsley & Garland, 2000). An evaluation of a person's SA, the
SAGAT is commonly employed in simulation-based experiments.
It comprises three levels: Level 1 SA (Perception), where people
can identify important environmental factors (such as objects,
ships, and traffic conditions in collision avoidance systems);
Level 2 SA (Comprehension), where they analyze this data to
evaluate risks and ascertain how the system will behave; and
Level 3 SA (Projection), where they forecast future results and
any interventions that may be required. Specific questions
regarding the scenario are posed when the participant's screen is
blanked out at different points throughout the simulation. The
SAGAT offers an unbiased evaluation of SA since the participant's
responses may be compared to the simulator's data (Endsley &
Garland, 2000). A set of thirty general SAGAT questions was
created for this experiment and dispersed throughout the various
traffic scenarios. These questions were then modified to better
suit the circumstances.

 Mental workload - The NASA-TLX was used to evaluate mental

workload (Hart & Staveland, 1988), which assesses six workload
dimensions: mental, physical, temporal, performance, effort, and
frustration level. Participants create weights for the dimensions
of this scale by comparing them pairwise. The overall workload
score is determined by adding the weighted workload scores for
each dimension. However, a version of the NASATLX was
employed in which participants only execute pairwise
 comparisons once, and only after all experimental trials have
been completed, because changing the weights after each run
takes some time and the task type remains constant during the
experiment. Therefore, all workload scores for each run were
subject to the weights obtained from the pairwise comparison.

 Task Performance – This is also known as time-to-

comprehension (TTC), was defined as the amount of time it took
participants to feel that they had a sufficient understanding of
the traffic situation through the information provided by the
CAGA system. Like other time-related performance measures,
like eye-tracking, reading speed, search time, and time to task
completion, TTC was selected as a measure of how quickly
humans can process information (Gawron, 2019). TTC was self-
guided and involved the participant in determining that the
traffic situation and the visualized solution were sufficiently
understood. The time measurement began when the traffic
situation was displayed and ended when the participant pressed
a key, after which the screen was blank. There is a time limit.
However, participants were encouraged to be as precise and fast
as they could.
Research gap:

The study tackles several current cognitive ergonomics issues,

especially when it comes to human supervision of self-governing marine
systems. It addresses the following major research gaps:

1. Static vs. Dynamic Environments

 Gap: Previous research, including this study, frequently

employs static or simplified scenarios, particularly in
maritime accident avoidance settings. Real-world maritime
scenarios are dynamic, with traffic patterns and conditions
continually shifting. This reduces the applicability of
findings from static situations to real-world operations.

 How the Study Addresses It: This study properly recognizes

its limitations. While it employs static radar images, it
encourages future research toward transparency in
dynamic settings utilizing real-time simulations. It
establishes the framework for future research by
investigating the impacts of transparency in a controlled,
although static, setting.

2. Practical Use of Transparency in Maritime Automation

 Gap: Although research on human-automation

collaboration has developed, there is no practical guidance
on how to integrate transparency elements in marine
collision avoidance systems to improve safety and
performance.
 How the Study Addresses It: By testing various levels of
openness in a CAGA system, the study sheds light on how
transparency might be structured to increase human
performance in future autonomous marine operations.
3. Absence of Multiple Task Demands

 Gap: The experiment centered on a single ship and task.

Real-world supervisors frequently supervise numerous
autonomous systems at the same time, forcing them to
divide their attention. This study does not address the
cognitive challenges of such multitasking situations.

 How the Study Addresses It: The paper acknowledges that

future operators will probably have to supervise many
ships. It specifically indicates that future research should
look into the impact of transparency on the supervisor's
capacity to notice and rectify performance discrepancies
when working under concurrent job demands. It
emphasizes the necessity for research that considers the
complexity of controlling many autonomous systems.

Research question:
The key research questions investigated in the study include:
1. What are the effects of agent transparency on the supervisor’s situation
awareness (SA)?
2. How does traffic complexity influence mental workload and task
performance?
3. Can increased transparency negate the negative effects of high traffic
complexity on SA and task performance?
Methodology:

What research design did the study use (e.g., experiment, survey, case
study, simulation)? What data collection methods were employed (e.g.,
eye tracking, EEG, task performance measure, self-support
questionnaire)? Who were the participants, and the study setting?

Research Design: The study used a repeated measure experimental

design, in which individuals completed all experimental conditions.

Independent Variables:

 Transparency: For this study, four levels of transparency

(Low, Medium A, Medium B, High) were established based
on the amount and type of information to be disclosed to a
supervisor when overseeing an autonomous Collison
Avoidance and Grounding Avoidance (CAGA) system.
  Traffic Complexity: Two levels of complexity were
defined for this study: traffic situations with low and high
complexity.

Dependent Variables:

 Situational Awareness (SA): measured using the Situation

Awareness Global Assessment Technique (SAGAT).

 Mental workload: measured using the NASA-TLX (modified

with a single pairwise comparison).

 Task performance: measured as time-to-comprehension

(TTC), which is the time it takes participants to understand
the circumstance and the system's recommended solution.

 Ranking: Participants assessed the four transparency levels

according to their observability and predictability.

Data Collection Methods:

 SAGAT Queries: Objective multiple-choice questions

assessing SA at different levels.
 NASA-TLX: Subjective workload ratings and pairwise
comparisons.
 Time-to-Comprehension (TTC): Objective measurement of
the time taken to understand the situation.
 Think-Aloud Protocol: Recording of participants' verbal
reasoning during the ranking task.
 Post-experiment semi-structured interviews: Recorded
using pen and paper.

Participants:

 For this study, 34 navigators with deck officer licenses (32

males and 2 females) were recruited as volunteers. Of these,
30 had an active license and 4 had navigation experience, but
their licenses had expired between 1 and 5 years before the
study.
Results and Key Findings

To summarize, the experiment's findings revealed that SA increased

with transparency, with level 1 SA highest in the high transparency
condition, level 2 highest in the medium (A) transparency condition, and
level 3 highest in the high transparency condition. For all SA measurements,
high-complexity traffic scenarios resulted in lower levels of SA. Furthermore,
there was no significant influence of transparency on mental workload, but a
substantial effect of complexity was discovered, indicating that higher traffic
complexity resulted in higher perceived mental workload. Furthermore, TTC
was highest at the middle (A) and high levels. TTC was also highest in the
most difficult traffic circumstances. Finally, the medium (A)- and high
transparency levels were rated as the most preferred by the participants.

The hypothesis about the favorable impacts of transparency on SA

were supported. However, the expected interaction effects between
transparency and complexity were not consistently observed.

Area of application

In which real-world domain (e.g., aviation, healthcare, automation) are the

findings applicable?

Deadline of Assignment: February 26, 2025 (5 PM)