Brazilian Journal of Technology 1

ISSN: 2595-5748

An empirical study on web API usability: The consumer-developer

perspective

Estudo empírico sobre a usabilidade de APIs web: A perspectiva dos

consumidores-desenvolvedores

Estudio empírico sobre la usabilidad de APIs web: La perspectiva de los

consumidores-desarrolladores
DOI:10.38152/bjtv7n4-014

Submitted: Oct 04th,, 2024

Approved: Oct 30th, 2024

Ariel Machini
Bachelor in Systems
Institution: Universidad Nacional de la Patagonia Austral
Address:Río Gallegos, Santa Cruz, Argentina
E-mail: amachini@conicet.gov.ar

Sandra Casas
Doctor in Systems
Institution: Universidade de Vigo (Spain)
Address:Río Gallegos, Santa Cruz, Argentina
E-mail: sicasas@uarg.unpa.edu.ar

ABSTRACT
The ease with which web APIs facilitate access to diverse resources and services has led to
their widespread adoption, consequently giving rise to a new business paradigm: the “API
economy”. Given this, usability has become a pivotal attribute for API adoption, and within
a competitive market, it merits detailed investigation. This exploratory study presents the
findings of a survey targeting both consumers and developers of web APIs. The primary
objectives of this survey were (1) to ascertain how respondents perceive various usability
factors and (2) to identify potential correlations between specific characteristics of
participant profiles and their expressed opinions. To fulfill these objectives, descriptive
statistical parameters and independence tests were employed. Results indicate that
documentation is regarded as the most influential factor on usability, while the quantity of
consecutive parameters of the same type is perceived as the least influential. Furthermore,
the findings reveal that expert opinions do not deviate significantly from those of non-
experts, and that a respondent's experience as a software developer and with web APIs can
influence their perceptions regarding certain factors.

Keywords: metrics, software quality, software libraries, human factors in software design.

RESUMO
A facilidade com que as APIs web permitem acessar diversos recursos e serviços levou à
sua ampla adoção, dando origem a um novo paradigma de negócios: a “economia API”.
Diante disso, a usabilidade tornou-se um atributo fundamental para a adoção de APIs e, em
um mercado competitivo, merece pesquisa detalhada. Este estudo exploratório apresenta os
resultados de uma pesquisa direcionada a consumidores e desenvolvedores de APIs web. Os
principais objetivos deste estudo foram (1) verificar como os entrevistados percebem

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 2
ISSN: 2595-5748

diversos fatores de usabilidade e (2) identificar possíveis correlações entre características

específicas dos perfis dos participantes e suas opiniões expressas. Para cumprir esses
objetivos, foram utilizados parâmetros estatísticos descritivos e testes de independência. Os
resultados indicam que a documentação é considerada o fator mais influente na usabilidade,
enquanto a quantidade de parâmetros consecutivos do mesmo tipo é percebida como o
menos influente. Finalmente, este trabalho revela que as opiniões de especialistas e não-
especialistas não divergem significativamente, e que a experiência de um entrevistado como
desenvolvedor de software e com APIs web pode influenciar suas percepções quanto a
determinados fatores.

Palavras-chave: métricas, qualidade de software, bibliotecas de software, fatores humanos

no desenho de software.

RESUMEN
La facilidad con la que las APIs web permiten acceder a diversos recursos y servicios ha
hecho que su utilización se masifique y, en consecuencia, surja una nueva perspectiva de
negocio, “la economía API”. Debido a esto, su usabilidad se vuelve una propiedad clave
para su adopción y, en un mercado competitivo, resulta de interés estudiarla en detalle. En
este trabajo, de carácter exploratorio, se presentan los resultados de una encuesta destinada
a consumidores y desarrolladores de APIs web, la cual tuvo como objetivos principales (1)
inquirir cómo los encuestados perciben distintos factores de usabilidad y (2) hallar posibles
relaciones entre ciertas características del perfil de los participantes y las opiniones que
dieron. Para cumplir con tales objetivos, se aplicaron algunos parámetros de la estadística
descriptiva y pruebas de independencia. Los resultados muestran que la documentación es
considerada como el factor con mayor influencia sobre la usabilidad y la cantidad de
parámetros consecutivos del mismo tipo como el factor menos influyente. Asimismo,
evidencian que la opinión de los expertos no varía demasiado de la de los demás, y que la
experiencia como desarrollador de software y con APIs web del encuestado puede influir
sobre la opinión que este tiene respecto a determinados factores.

Palabras clave: métricas, calidad de software, librerías de software, factores humanos en

el diseño de software.

1 INTRODUCTION

Over the past decade, web APIs (Application Programming Interfaces) have become
a cornerstone of modern application development (Raemaekers; van Deursen; Visser, 2012).
Beyond fostering software reuse and enabling access to resources and services from and
between organizations, web APIs have given rise to a new business paradigm known as the
“API economy” (Tan et al., 2016). Consequently, the web API market has become highly
competitive, making usability a pivotal attribute for determining an API's value (Stylos;
Myers, 2007).
Usability is considered one of the most important quality attributes of software
(Nielsen, 1992), and while there are various definitions, the one provided by ISO 9241-11
is perhaps the most well-known: “The extent to which a product can be used by specified

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 3
ISSN: 2595-5748

users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified
context of use”.
To better understand their users, their opinions, and areas for improvement, leading
companies in the web API sector such as RapidAPI and SmartBear conduct large-scale
surveys on a regular basis. However, while these surveys address various topics (such as the
overall API industry, programming languages and technologies used, and testing), and
although usability is among them, the questions included regarding usability tend to be
oriented towards market needs or specifically focused on documentation. For that reason,
between September 27, 2023, and December 1 of the same year, we conducted an
exploratory survey focused on web API usability to gather the opinions of API users—the
developers—regarding various factors that could influence usability. This survey was
designed to answer the following questions:
Q1: What factors have the greatest influence over the usability of web APIs, from
the perspective of consumers and/or developers of web APIs?
Q2: How does the use of specifications for the documentation of web APIs influence
usability?
Q3: How does the level of expertise of the respondent influence their opinions?
The survey was completed by 36 individuals, most of whom are Spanish-speaking
developers. The results suggest that the documentation of the elements of a web API is the
factor with the greatest influence on usability, and that the number of consecutive
parameters of the same type is the factor that has the least influence. Additionally, the results
suggest that opinions from experts do not differ significantly from those of other survey
participants, and that a respondent's experience as a software developer and with web APIs
can influence their opinions regarding certain factors.
This study is structured as follows: Section 2 describes work related to the topic
addressed in this study; Section 3 explains the methodology followed for its development;
Section 4 presents the results of analyzing the survey; Section 5 addresses threats to validity;
and Section 6 discusses the significance of these findings.

2 THEORETICAL FRAMEWORK

In this section, we present various studies, both from academia and the web API
industry, that are related to the topic addressed in our work and involve surveys or
interviews. All these studies were ordered in descending order by publication year.
Postman surveys (Postman, 2023) – Postman conducts annual surveys exploring

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 4
ISSN: 2595-5748

development priorities, API tools, and future API visions. These surveys gather responses
from over 40 thousand developers and API professionals, ranging from CEOs and Customer
Success Managers to full-stack developers. Based on the collected data, Postman publishes
freely accessible reports titled “State of the API Report” on its website. These reports
include graphs summarizing the information, and present key findings. Seven notable
findings emerged from the 2023 Postman survey: (1) APIs are a significant source of
revenue for most providers; (2) Price is an increasingly important factor when choosing an
API; (3) Investments in APIs are expected to increase or remain stable; (4) Most API
professionals utilize AI to streamline coding or detect bugs; (5) There was a surge in
respondents self-identifying as “API-first leaders”; (6) Departing API developers leave a
substantial gap (highlighted issues include outdated documentation and zombie APIs); and
(7) While API security is improving, it remains an area requiring further attention.
SmartBear surveys (SmartBear, 2023) – SmartBear, the company behind Swagger,
regularly publishes reports known as the “State of Software Quality: API” that include,
among other things, information on future trends and the evolution of APIs. To compile
these reports, SmartBear conducts surveys involving over a thousand API users worldwide
across 17 different industries. Most participants in these surveys are end-users of Swagger,
SoapUI, and ReadyAPI. Results from the 2023 survey highlight that (1) Event-driven and
messaging APIs are on the rise; (2) AI is emerging as the new driving force behind API
growth; (3) Documentation is being given increased importance; (4) Standardization and
security remain challenges that organizations are seeking to address; and (5) Formal testing
processes are becoming more prevalent.
An empirical study of API Management and ISO/IEC SQuaRE: a practitioners’
perspective (dos Santos; Casas, 2024) – In this work, the authors designed a structured
survey using the ISO/IEC 25010 (SQuaRE) standard as a reference model. This survey was
presented to professional developers, system administrators, and software functional
analysts to understand how they perceive quality characteristics related to API management
functions. After conducting a statistical analysis of the responses, the authors concluded that
results indicate that functional suitability and security are perceived as the most critical
quality capabilities, and commented that their findings could serve as a foundation for future
research.
RapidAPI surveys (RapidAPI, 2022) – Every year, RapidAPI (self-proclaimed as
“The world's largest API marketplace”) conducts a developer survey known as “State of
APIs” to uncover changes in industry trends, developer experiences with different API
types, the kinds of APIs consumed by various companies, and more. The survey typically

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 5
ISSN: 2595-5748

runs for a month and gathers responses from developers in over 100 countries. Regarding
the results of the 2022 survey (the most recent at the time of writing), it is noteworthy that
(1) TypeScript is preferred over PHP for building APIs; (2) Approximately 65 % of
developers reported greater API dependency in 2022 compared to 2021; (3) Partner APIs
saw a ~10 % increase in the technology sector; (4) The use of REST APIs increased by 10
% in 2022; and (5) API monetization grew by 16 % in the financial sector.
Usos y problemas de las APIs Web en la República Argentina (Constanzo et al.,
2022) – The authors of this study conducted a survey aimed at understanding the usage and
challenges faced by software developers consuming web APIs in Argentina. Their findings
reveal that the most critical issues center around changes due to the evolution of APIs and
access to answers to questions that these changes imply. In terms of usability, challenges
such as missing or poor documentation and the predominance of manual problem-solving
were highlighted.
From Code Refactoring to API Refactoring: Agile Service Design and Evolution
(Stocker; Zimmermann, 2021) – In this study, Stocker and Zimmermann present the results
of a survey targeted to web API providers, carried out to understand the reasons that lead
software architects and API developers to make changes, as well as the consequences of
these changes (similar to the investigation of Espinha, Zaidman and Gross (2014)). The
survey involved 64 professionals, who were contacted personally by the authors or found
on social media. Based on their results, the researchers concluded that (1) changes to APIs
are primarily caused by modifications in functional requirements, driven by either the client
or the provider, and that (2) the second reason for the occurrence of changes is non-
functional requirements, mainly quality issues. Usability and maintainability, followed by
performance and scalability, are the quality attributes that drive the most changes. They also
concluded that (3) regarding change and mitigation strategies applied by API providers, the
specification or code is updated, documentation is adjusted, versioning is used, and clients
are adapted.
You Broke My Code: Understanding the Motivations for Breaking Changes in
APIs (Brito et al., 2019) – In order to understand the reasons behind the introduction of
breaking changes in APIs, the authors of this study conducted a four-month field study on a
set of well-known local APIs. Subsequently, the developers of these APIs were surveyed to
know their motivations for implementing such changes. The results of this study show that
breaking changes are primarily driven by the need to introduce new functionalities or to
simplify APIs, and to improve maintainability. Additionally, a complementary study found
that these types of changes have a significant impact on API clients, and concluded with a

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 6
ISSN: 2595-5748

series of recommendations.
An Exploratory Study on Faults in Web API Integration in a Large-Scale Payment
Company (Aué et al., 2018) – In this study, the authors investigated the impact of errors on
APIs and analyzed over 2 million API error responses to help reduce the impact of API-
related issues by identifying underlying failures. Subsequently, a survey targeted to API
consumers was conducted to validate the quantitative results obtained. Their findings reveal
that (1) API integration failures can be categorized into 11 general causes; (2) the majority
of failures can be attributed to one of these 11 categories (invalid or missing request data)
and (3) the lack of implementation and integration guidelines, as well as error recovery
guidance, poses a significant challenge for developers.
How API Documentation Fails (Uddin; Robillard, 2015) – In this study, authors
investigated how 10 common documentation problems manifest in practice. They conducted
two surveys, with a total of 323 participants (professional software developers), in which
they analyzed 179 documentation units. The results identified ambiguity, incompleteness,
and incorrectness of content as the most severe problems; additionally, participants
indicated that 6 of the 10 documentation problems considered in this study were reason
enough to seek alternative APIs.
Web API growing pains: Stories from client developers and their code (Espinha;
Zaidman; Gross, 2014) – Due to the complications experienced by developers as a result of
the evolution of the web APIs they use, the authors of this article conducted a semi-
structured survey of developers who use web APIs in their work, to better understand these
complications. They also investigated how major web API providers organize the evolution
of their APIs and how these changes affect the source code of their clients. Finally, they
comment that their results highlight the impact of service evolution on client software and
propose a set of observations regarding best practices for the evolution of web APIs.
An Empirical Study of API Usability (Piccioni; Furia; Meyer, 2013) – In this work,
focused on local API usability, the authors present a study with 25 programmers that
combined interviews with systematic observations of programmers' behavior in developing
specific programming tasks. According to the authors, the results reinforce data presented
by similar studies, such as the difficulty of finding good names for API functionalities, and
also revealed new problems related to API design, such as the impact of flexibility and the
accuracy of documentation.

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 7
ISSN: 2595-5748

3 METHODOLOGY

This research focuses on the development of a survey aimed at web API consumers
and/or developers, conducted with the purpose of understanding which factors have the
greatest influence on the usability of these kind of APIs, as analyzed from the perspective
of the respondents. According to the classification of survey purposes given by Wohlin et
al. (2012), the survey developed within the framework of this study is exploratory.
A non-probabilistic sampling scheme (specifically, convenience sampling), was
employed, where participant selection is done manually, having in consideration their
representativeness and accessibility (Fowler, 2013; Kasunic, 2005). We used this method
because we had time constraints and we did not have access to a list of people (of interest
for our study) to use as a sampling frame. To find interested individuals with the required
profile, we searched websites related to software development and/or web APIs, and shared
the survey on forums and social media. To be more precise, the survey was sent to over 500
email addresses extracted from the Cámara de la Industria Argentina de Software (CESSI) 1;
it was published on the Postman2 and API Craft3 forums; and it was also shared on LinkedIn,
in Facebook groups on computer science and systems, and on Twitter (with specialized web
API groups). In all cases, it was made explicit that developers with prior experience with
web APIs were being sought.
The survey consisted of a total of 43 questions, with the first 7 collecting information
about the respondent's profile, the following 35 referring to the influence of different factors
on the usability of web APIs, and the last one for sharing suggestions. All questions
regarding web API usability were closed-ended, in which the respondent had to select a
value between 1 and 5 (Likert scale) to determine how much they believe the factor in
question can influence the usability of a web API. The factors included in the questions were
extracted from scientific articles (Koçi et al., 2020; Ma et al., 2016; Mosqueira-Rey et al.,
2018; Scheller; Kühn, 2015; de Souza; Bentolila, 2009; Rama; Kak, 2013) found through a
bibliographic analysis related to web API usability, from blogs (MuleSoft, n.d.; Riggins,
2015; Levin, 2019; Au-Yeung; Donovan, 2020), reports (Hämäläinen, 2019), and design
guidelines (Mulloy, 2012) published by sectors of the web API industry. The survey was
open from the end of 2023 September to December 1 of the same year.
Descriptive statistics such as mean, median, and standard deviation were used to

1
https://cessi.org.ar/socios
2
https://community.postman.com
3
https://groups.google.com/g/api-craft

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 8
ISSN: 2595-5748

obtain basic information about the collected data. Additionally, Chi-Square tests of
independence (or Fisher's exact tests in cases where the Chi-Square test could not be
applied) were conducted to study a possible link between certain variables of interest and
the level of influence that respondents assigned to the factors covered by the survey (all
categorical variables). To form the contingency tables, the categories of most variables were
combined so that 2x2 tables could be constructed (in order to be able to resort to Fisher's
exact test when the assumptions for using the Chi-Square test were not met). All hypothesis
tests were conducted using a significance level (α) of 0.05.
To elaborate the questionnaire, Google Forms, a free and easy to use tool that allows
creating forms of varying complexity, was used. This tool also provides functionality for
extracting and organizing responses, to facilitate their analysis. Additionally, InfoStat4, a
statistical software developed by an Argentine team of researchers specializing in applied
statistics, was used to perform a variety of statistical analyses on the collected data.

4 RESULTS AND DISCUSSIONS

During the period in which the survey was open, a total of 36 responses were
received, of which 34 came from Spanish speakers. The majority of the people who
responded to the survey (88.9 %) are residents of Argentina, but there were also responses
from residents of Uruguay (5.7 %), the United States (2.7 %), and Germany (2.7 %). A large
proportion of respondents (86.1 %) work in the private sector or are self-employed (30.6
%), however, there are also those who work in the academic sector (8.3 %) and public sector
(5.6 %). This information is graphically represented in Figure 1. Following a similar line,
with regard to the areas of expertise (Figure 2), 80.6 % of participants are back-end
developers, 63.9 % work with databases, 55.6 % are front-end developers, 36.1 % mobile
application developers, 33.3 % desktop application developers, 27.8 % work in the cloud
computing area, 25 % as DevOps, and 2.8 % as data scientists. It is important to clarify that,
in both the work sector and area of expertise, the respondents were allowed to choose more
than one option.

4
https://www.infostat.com.ar

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 9
ISSN: 2595-5748

Figure 1. Distribution of respondents between different work sectors.

Source: Prepared by the authors themselves.

Figure 2. Distribution of respondents between different areas of work.

Source: Prepared by the authors themselves.

Regarding the participants' experience as software developers (Figure 3), 36.1 %

indicated that they have an intermediate level and 63.9 % reported being experts. As for
their experience as web API users (Figure 4), the majority of respondents (38.9 %) indicated
that they have 5 to 10 years of experience, followed by those who indicated they have 3 to
5 years of experience (36.1 %), more than 10 years (19.5 %), and less than 3 years (5.5 %).

Figure 3. Respondents’ level of experience developing software in general.

Source: Prepared by the authors themselves.

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 10
ISSN: 2595-5748

Figure 4. Respondents’ level of experience with web APIs.

Source: Prepared by the authors themselves.

Lastly, regarding experience with web APIs themselves (Figure 5), the vast majority
of respondents (75 %) indicated that they both consume and develop web APIs, 16.7 % only
consume web APIs, and the remaining 8.3 % only develop web APIs. As illustrated in
Figure 6, 91.7 % of participants reported using external web APIs, 86.1 % internal web
APIs, and 44.4 % partner web APIs. In this case, respondents were allowed to select more
than one option.

Figure 5. Whether the respondents consume and/or develop web APIs.

Source: Prepared by the authors themselves.

Figure 6. Types of web APIs used by the respondents.

Source: Prepared by the authors themselves.

Continuing our analysis on the degree of influence that respondents attributed to each
of the factors considered in this study, Table 1 presents a set of descriptive statistics. These
include measures of central tendency such as the mean, as well as measures of dispersion
such as the standard deviation, minimum value, and maximum value. Through these
indicators, we aim to gain a comprehensive understanding of participants' perceptions

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 11
ISSN: 2595-5748

regarding each of the aforementioned factors.

Table 1. Summary statistics.

Factor Mean Median SD Min Max
Clarity of names 4.4 5 0.84 1 5
Data type specificity 4.2 4 1.00 1 5
Completion of declared task 4.6 5 0.68 3 5
Number of endpoints 3.0 3 1.21 1 5
Number of parameters 3.6 4 1.08 1 5
Number of return values 3.4 4 1.32 1 5
Response complexity 3.1 3 1.38 1 5
Innapropriate String usage 3.6 4 1.08 1 5
Verbosity of error messages 3.7 4 1.26 1 5
Versioning 3.8 3 0.87 3 5
JSON support 4.1 4 0.89 3 5
Support for multiple return formats 3.1 3 1.17 1 5
OAuth usage 3.4 3 0.99 1 5
Proper usage of response headers 3.8 4 0.85 3 5
Developer Stack Size 3.6 3 1.08 1 5
Usage of verbs in base URLs 3.0 3 1.08 1 5
Usage of plural nouns for resource names 3.2 3 1.27 1 5
Specificity for resource naming 3.5 3 1.08 1 5
Usage of nearly-identical names for new versions of
3.8 4 1.27 1 5
an endpoint
Endpoint name similarity 4.0 4 1.16 1 5
Documentation of API elements 4.8 5 0.40 4 5
Documentation - Identification of deprecated
4.4 5 0.90 1 5
elements
Documentation - Inclusion of usage examples 4.5 5 0.65 3 5
Documentation - Inclusion of error information 4.4 5 0.69 3 5
Documentation - Specification usage 3.9 4 1.15 1 5
Availability of additional help sources 3.4 3 0.99 2 5
Consecutive parameters of the same type 2.2 2 1.30 1 5
Support for filtering, pagination and sorting 3.9 4 0.95 2 5
Task-Invocation Ratio 3.8 4 0.91 3 5
Path depth 3.0 3 1.23 1 5
Name consistency 4.3 4 0.91 1 5
Efficiency 4.2 5 1.13 1 5
Security 4.4 5 0.97 1 5
Errors 4.4 5 1.02 1 5
Price 3.8 4 1.31 1 5
Source: Prepared by the authors themselves.

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 12
ISSN: 2595-5748

As shown in Table 1, with the exception of Consecutive parameters of the same type,
all factors had mean and median scores of 3 or higher (on a 1-5 scale). Additionally, the
minimum values column highlights several factors with minimum scores of 3 or even 4 (of
influence over usability).
With respect to the analysis of the relationship between certain variables and the
degree of influence assigned to the survey factors, the variables of interest were: (1) the
level of experience as a software developer, and (2) the years of experience with web APIs.
To conduct this analysis, a large number (70) of contingency tables were built. Table 2
presents the instances where the p-value of the statistic used indicated a relationship or
dependency between the variables.

Table 2. Cases where a relationship between variables was found

(p-value < α).
Variable 1: Experience as a software developer
Disagree/
Factor Agree p-value
Neutral
Usage of plural nouns for Intermediate 8 5
0,0298
resource names Expert 5 18
Specificity for resource Intermediate 11 2
0,0007
naming Expert 6 17
Intermediate 11 2
Price 0,0162
Expert 10 13
Variable 2: Experience with web APIs
Disagree/
Factor Agree p-value
Neutral
<5 years 3 11
OAuth usage 0,0494
≥5 years 11 10
<5 years 12 3
Price 0,0258
≥5 years 9 12
Source: Prepared by the authors themselves (using InfoStat).

The p-values in bold are the cases where Fisher's exact test was applied due to small
sample sizes or low expected counts.
Tables 3 and 4 show the disparities in responses between the experienced and less
experienced respondents.

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 13
ISSN: 2595-5748

Table 3. Differences between the opinions of expert and least experienced respondents (Mean).
Factor Expert mean Non-expert mean Difference
Clarity of names 4.48 4.27 0.21
Data type specificity 4.29 4.00 0.29
Completion of declared task 4.48 4.87 -0.39
Number of endpoints 2.81 3.20 -0.39
Number of parameters 3.57 3.60 -0.03
Number of return values 3.48 3.33 0.15
Response complexity 3.05 3.13 -0.08
Innapropriate String usage 3.57 3.60 -0.03
Verbosity of error messages 3.71 3.67 0.04
Versioning 3.71 3.80 -0.09
JSON support 4.00 4.13 -0.13
Support for multiple return
3.24 2.80 0.44
formats
OAuth usage 3.57 3.07 0.50
Proper usage of response headers 3.90 3.73 0.17
Developer Stack Size 3.62 3.53 0.09
Usage of verbs in base URLs 3.14 2.73 0.41
Usage of plural nouns for resource
3.14 3.33 -0.19
names
Specificity for resource naming 3.29 3.80 -0.51
Usage of nearly-identical names
3.81 3.73 0.08
for new versions of an endpoint
Endpoint name similarity 3.90 4.07 -0.17
Documentation of API elements 4.81 4.80 0.01
Documentation - Identification of
4.48 4.27 0.21
deprecated elements
Documentation - Inclusion of
4.29 4.80 -0.51
usage examples
Documentation - Inclusion of
4.52 4.33 0.19
error information
Documentation - Specification
4.24 3.53 0.71
usage
Availability of additional help
3.29 3.47 -0.18
sources
Consecutive parameters of the
2.10 2.27 -0.17
same type
Support for filtering, pagination
3.95 3.93 0.02
and sorting
Task-Invocation Ratio 3.81 3.87 -0.06
Path depth 3.19 2.67 0.52
Name consistency 4.14 4.40 -0.26
Efficiency 4.19 4.13 0.06
Security 4.33 4.53 -0.20
Errors 4.57 4.13 0.44

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 14
ISSN: 2595-5748

Price 3.29 4.47 -1.18

Source: Prepared by the authors themselves.

Table 4. Differences between the opinions of expert and least experienced respondents (Median).
Factor Expert median Non-expert median Difference
Clarity of names 5 5 0
Data type specificity 4 4 0
Completion of declared task 5 5 0
Number of endpoints 3 3 0
Number of parameters 4 4 0
Number of return values 4 4 0
Response complexity 3 3 0
Innapropriate String usage 4 3 1
Verbosity of error messages 4 3 1
Versioning 3 4 -1
JSON support 4 4 0
Support for multiple return
3 3 0
formats
OAuth usage 4 3 1
Proper usage of response headers 4 3 1
Developer Stack Size 4 3 1
Usage of verbs in base URLs 3 3 0
Usage of plural nouns for resource
3 3 0
names
Specificity for resource naming 3 4 -1
Usage of nearly-identical names
4 4 0
for new versions of an endpoint
Endpoint name similarity 4 4 0
Documentation of API elements 5 5 0
Documentation - Identification of
5 5 0
deprecated elements
Documentation - Inclusion of
4 5 -1
usage examples
Documentation - Inclusion of
5 5 0
error information
Documentation - Specification
4 4 0
usage
Availability of additional help
3 3 0
sources
Consecutive parameters of the
2 2 0
same type
Support for filtering, pagination
4 4 0
and sorting
Task-Invocation Ratio 4 3 1
Path depth 3 2 1
Name consistency 4 5 -1

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 15
ISSN: 2595-5748

Efficiency 5 5 0
Security 5 5 0
Errors 5 5 0
Price 3 5 -2
Source: Prepared by the authors themselves.

Analysis of the survey data reveals that respondents possess a considerable level of
experience (intermediate to high). Furthermore, contrary to expectations, the opinions of
web API experts and the rest of the respondents do not vary significantly (as seen in Tables
3 and 4), addressing with this the research question Q3. The most notable difference was
observed in the Price factor; less experienced respondents considered price to have a
significant impact on usability (mean: 4.47; median: 5), while experts held a more moderate
view (mean: 3.29; median: 3).
Regarding research question Q1, according to the participants, the factor
Documentation of API elements, is the one that has the greatest influence on usability (SD:
0.4; min: 4; mean: 4.8; median: 5). This is in line with what is expressed in (SmartBear,
2023; Uddin; Robillard, 2015; Mosqueira-Rey et al., 2018; Rama; Kak, 2013; MuleSoft,
n.d.), where the criticality of including complete and exhaustive documentation along with
the API is emphasized. On the other hand, and despite being considered as an important
factor in (Scheller; Kühn, 2015; Rama; Kak, 2013), participants voted as least influential
the factor Consecutive parameters of the same type (SD: 1.3; min: 1; mean: 2.2; median: 2).
Other factors considered as very influential by survey participants are: Completion of
declared task (SD: 0.68; min: 3; mean: 4.6; median: 5); Inclusion of usage examples in the
documentation (SD: 0.65; min: 3; mean: 4.5; median: 5); Inclusion of error information in
the documentation (SD: 0.69; min: 3; mean: 4.4; median: 5); Clarity of names (SD: 0.84;
min: 1; mean: 4.4; median: 5); Identification of deprecated elements in the documentation
(SD: 0.9; min: 1; mean: 4.4; median: 5); Security (SD: 0.97; min: 1; mean: 4.4; median: 5);
Errors (SD: 1.02; min: 1; mean: 4.4; median: 5) and Efficiency (SD: 1.31; min: 1; mean:
4.2; median: 5).
All listed factors have a median of 5 (the highest possible value), and they were
ordered by the difference between their median and mean scores. A larger difference
indicates greater standard deviations and lower minimum values chosen by respondents.
Additionally, and addressing research question Q2, even though its median is not 5,
participants also considered Documentation – Specification usage to be important (SD: 1.15;
min: 1; mean: 3.9; median: 4) for usability.
The independence tests conducted indicated several relationships. Specifically, the

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 16
ISSN: 2595-5748

respondent's experience as a software developer was found to be associated with the

following factors:
• Usage of plural nouns for resource names: 78.3 % of experts gave an influence value
≤ 3 on the Likert scale to this factor. In contrast, only 38.5 % of the respondents with
an intermediate level of experience gave an influence value ≤ 3 to this factor. In
conclusion: Those with more experience developing software in general consider
that using plural nouns to name resources has a lesser influence on web API
usability;
• Specificity for resource naming: In this case, the contingency table revealed a notable
contrast. While 73.9 % of experts rated this factor ≤ 3 on the Likert scale, 84.6 % of
intermediate-level developers rated it > 3. In conclusion: Those with more
experience developing software in general consider that the specificity of the names
used for resources has a lesser influence on the usability of web APIs;
• Price: While 56.5 % of experts gave an influence value ≤ 3 to this factor, a
significantly higher 84.6 % of intermediate-level developers gave it a value of > 3 in
the scale. In conclusion: Those with less experience developing software in general
consider that the cost of use of a web API strongly influences on its usability.
When considering the respondent's experience with web APIs, a dependency was
found with two factors:
• OAuth usage: Most experienced participants (≥ 5 years of experiencie) had divided
opinions (52.4 % gave this factor a value of influence > 3 in the scale); In contrast,
most (80 %) of respondents with less than 5 years of experience with web APIs
considered that this factor does not play an important role in web API usability. This
difference is interesting, since this particular factor is highly related with web APIs
(more specifically, with their security: OAuth is a widely known authorization
protocol) and, even though the opinions of the experts were divided, the percentage
that gave a value of 4 or 5 in the scale to this factor is slightly higher to the percentage
that gave a value ≤ 3. To summarize, developers more experienced web APIs tend
to consider that this factor has a greater influence over their usability;
• Price: Similar to the interaction Price-Experience as a software developer, in this
case it also happened that most (57.1 %) of those who claimed to have more
experience with web APIs (≥ 5 years) gave a lower influence value (≤ 3) on the scale
to the factor Price, while those with less experience (< 5 years) mostly
(80 %) gave values > 3 to the aforementioned factor. This likeness in the observed
percentages serves to reinforce the assumption that the level of experience of the

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 17
ISSN: 2595-5748

user may affect their perception of the influence of the cost of use of a web API over
its usability.

5 CONCLUSION

Although our findings are based on a preliminary exploration and thus require further
validation to be generalized, they align with the results of several related studies. For
instance, Postman (2023) highlights that respondents also consider the price of a web API
as an important factor. Similarly, the SmartBear (2023) survey results emphasize the
significance of documentation and security. In 2, participants indicated that documentation
is a paramount factor, and poor documentation is sufficient reason to seek alternative APIs.
This exploratory study will then enable us to formulate new hypotheses for future research
on web API quality and usability. As for the future work, we plan to develop a usability
model for web APIs that incorporates the influence values assigned by respondents to the
different factors, thereby facilitating the evaluation of the usability of such APIs.

ACKNOWLEDGEMENTS

We would like to thank CONICET for funding our investigation and the project “Mejora y
evaluación de la usabilidad de APIs web”.

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 18
ISSN: 2595-5748

REFERENCES

AU-YEUNG, J; DONOVAN, R. Best practices for REST API design, Mar. 2020.
Retrieved from: https://stackoverflow.blog/2020/03/02/best-practices-for-rest-api-design

AUÉ, J.; ANICHE, M.; LOBBEZOO, M.; VAN DEURSEN, A. An exploratory study on
faults in web API integration in a large-scale payment company. In: Proceedings of the
40th International Conference on Software Engineering: Software Engineering in
Practice, May 2018. doi:10.1145/3183519.3183537

BRITO, A.; VALENTE, M. T.; XAVIER, L.; HORA, A. You broke my code:
Understanding the motivations for breaking changes in APIs. Empirical Software
Engineering, vol. 25, no. 2, Nov. 2019. doi:10.1007/s10664-019-09756-z

CONSTANZO, M.; CASAS, S.; VIDAL, G.; CRUZ, D. Usos y problemas de las APIs
Web en la República Argentina. Revista Tecnología y Ciencia, no. 44, Jul. 2022.
doi:10.33414/rtyc.44.79-97.2022

DE SOUZA, C. R.; BENTOLILA, D. L. Automatic evaluation of API usability using

complexity metrics and visualizations. In: 2009 31st International Conference on
Software Engineering - Companion Volume, May 2009. doi:10.1109/ICSE-
COMPANION.2009.5071006

DOS SANTOS, E.; CASAS, S. An empirical study of API Management and ISO/IEC
SQuaRE: a practitioners’ perspective. In: Libro de Actas: XXIX Congreso Argentino de
Ciencias de la Computación. Luján: 1st ed., 2024.

ESPINHA, T.; ZAIDMAN, A.; GROSS, H.-G. Web API growing pains: Stories from
client developers and their code. In: 2014 Software Evolution Week - IEEE Conference
on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE),
Feb. 2014. doi:10.1109/csmr-wcre.2014.6747228

FOWLER, F. J. Survey Research Methods. London: SAGE Publications, 2013.

GHAZI, A. N.; PETERSEN, K.; REDDY, S. S.; NEKKANTI, H. Survey Research in

Software Engineering: Problems and Mitigation Strategies. IEEE Access, vol. 7, 2019.
doi:10.1109/ACCESS.2018.2881041

HÄMÄLÄINEN, O. API-First Design with Modern Tools. 2019. 57 pages. Bachelor’s

thesis (Business Information Technology) - JAMK University of Applied Sciences,
Finland, 2019. Available in:
https://www.theseus.fi/bitstream/handle/10024/226493/Hamalainen_Oona.pdf

KASUNIC, M. Designing an Effective Survey. Carnegie Mellon University, Ago. 2005.

doi: 10.1184/R1/6573062.v1

KOÇI, R.; FRANCH, X.; JOVANOVIC, P.; ABELLÓ, A. A Data-Driven Approach to

Measure the Usability of Web APIs. In: 2020 46th Euromicro Conference on Software
Engineering and Advanced Applications (SEAA), Ago. 2020.
doi:10.1109/SEAA51224.2020.00021

LEVIN, G. Top 5 REST API Design Problems. Dec. 2019. Retrieved from

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 19
ISSN: 2595-5748

https://blog.restcase.com/top-5-rest-api-design-problems

MA, S.-P.; LAN, C.-W.; HO, C.-T.; YE, J.-H. QoS-Aware Selection of Web APIs Based
on ε-Pareto Genetic Algorithm. In: 2016 International Computer Symposium (ICS),
Dic. 2016. doi:10.1109/ICS.2016.0122

MOSQUEIRA-REY, E.; ALONSO-RÍOS, D.; MORET-BONILLO, V.; FERNÁNDEZ-

VARELA, I.; ÁLVAREZ-ESTÉVEZ, D. A systematic approach to API usability:
Taxonomy-derived criteria and a case study. Information and Software Technology, vol.
97, May 2018. doi:10.1016/j.infsof.2017.12.010

MULESOFT. Four Principles for Designing Effective APIs. Retrieved from:

https://www.mulesoft.com/api-university/four-principles-designing-effective-apis

MULLOY, B. Web API Design. Apigee, 2012.

NIELSEN, J. The usability engineering life cycle. Computer, vol. 25, no. 3, Mar. 1992.
doi:10.1109/2.121503

PICCIONI, M.; FURIA, C. A.; MEYER, B. An empirical study of API usability. In: 2013
ACM / IEEE International Symposium on Empirical Software Engineering and
Measurement, Oct. 2013. doi:10.1109/esem.2013.14

POSTMAN. 2023 State of the API Report. Retrieved from:

https://www.postman.com/state-of-api

RAEMAEKERS, S.; VAN DEURSEN, A.; VISSER, J. Measuring software library

stability through historical version analysis. In: 2012 28th IEEE International
Conference on Software Maintenance (ICSM), Sep. 2012.
doi:10.1109/icsm.2012.6405296

RAMA, G. M.; KAK, A. Some structural measures of API usability. Software: Practice
and Experience, vol. 45, no. 1, Sep. 2013. doi:10.1002/spe.2215

RAPIDAPI. State of APIs 2022. Retrieved from https://stateofapis.com

RIGGINS, J. Why API Developer Experience Matters More Than Ever. Mar. 2015.
Retrieved from https://nordicapis.com/why-api-developer-experience-matters-more-than-
ever

SCHELLER, T.; KÜHN, E. Automated measurement of API usability: The API Concepts
Framework. Information and Software Technology, vol. 61, May 2015.
doi:10.1016/j.infsof.2015.01.009

SMARTBEAR. 2023 State of Software Quality: API. Retrieved from

https://smartbear.com/state-of-software-quality/api

STOCKER, M.; ZIMMERMANN, O. From code refactoring to API refactoring: Agile

service design and evolution. In: Service-Oriented Computing, 2021. doi:10.1007/978-3-
030-87568-8\_11

STYLOS, J.; MYERS, B. Mapping the space of API design decisions. In: IEEE

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024

 Brazilian Journal of Technology 20
ISSN: 2595-5748

Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2007),

Sep. 2007. doi:10.1109/vlhcc.2007.44

TAN, W.; FAN, Y.; GHONEIM, A.; HOSSAIN, M. A.; DUSTDAR, S. From the service-
oriented architecture to the web API economy. IEEE Internet Computing, vol. 20, no. 4,
Jul. 2016. doi:10.1109/mic.2016.74

UDDIN, G.; ROBILLARD, M. P. How API documentation fails. IEEE Software, vol. 32,
no. 4, Jul. 2015. doi:10.1109/ms.2014.80

WOHLIN, C. et al. Experimentation in Software Engineering. 1st ed. Heidelberg:

Springer Berlín, 2012. doi:10.1007/978-3-642-29044-2

Brazilian Journal of Technology, Curitiba, v.7, n.4, p. 01-20, 2024