Contents
Introduction ............................................................................................................................................................ 2
Description of the Application................................................................................................................................ 2
Raffle Creator Interface ...................................................................................................................................... 2
Home Interface, Navigation and Additional Requirements ............................................................................... 2
Raffle Manager Interface ................................................................................................................................... 3
Ticket Selling & Listing Interface(s) .................................................................................................................. 3
Show Winner Interface....................................................................................................................................... 4
Testing Methodology .............................................................................................................................................. 4
Test User Details ................................................................................................................................................ 4
Preparation Tasks ............................................................................................................................................... 4
Introductory Material ......................................................................................................................................... 4
Test Cycle 1: Project 1 Prototype 1 of Mirza Istiaque Ahmed ........................................................................... 4
Tasks for Test User Number 1: Create a raffle and edit raffle name ............................................................. 4
Task for Test User Number 1: Draw a winner for a given raffle ................................................................... 5
Task for Test User Number 1: Sell tickets for raffle and edit customer name for ticket ................................ 5
Test Cycle 2: Project 1 Prototype 2 of Mirza Istiaque Ahmed ........................................................................... 5
Tasks for Test User Number 2: Create a raffle and edit raffle name ............................................................. 5
Task for Test User Number 2: Draw a winner for a given raffle ................................................................... 5
Task for Test User Number 2: Sell tickets for raffle and edit customer name for ticket................................ 5
Test Cycle 3: Project 1 Prototype 3 of Mirza Istiaque Ahmed ........................................................................... 5
Task for Test User Number 3: Create a raffle and edit raffle name ............................................................... 5
Task for Test User Number 3: Draw a winner for a given raffle ................................................................... 6
Task for Test User Number 3: Sell tickets for raffle and edit customer name for ticket ................................ 6
Questionnaire ................................................................................................................................................. 6
Test Results ............................................................................................................................................................ 6
Completion Time results .................................................................................................................................... 6
Ease of use results .............................................................................................................................................. 6
Intervention Results ........................................................................................................................................... 7
Discussion............................................................................................................................................................... 7
Conclusion .............................................................................................................................................................. 7
References .............................................................................................................................................................. 8
Introduction
Usability studies have been utilized to test different products and computer software (Nelson, 2011). Given the
current pandemic situations, the tests were carried out using a video conference setting. The general opinion
about the online usability test identifies similar errors as the usability test carried out in lab settings (Tullis,
Fleischman, McNulty, Cianchette, & Bergel, 2002).
The report focuses extensively on the think-aloud usability test cycles to understand the mental models of the
end-users of the raffle draw application prototypes presented in the first project. The report includes a
description of the functionalities of the raffle draw application. It incorporates a result and discussion section to
further the finding and identify limitations that can be tackled during the future development of the raffle draw
application.
Description of the Application
The native raffle draw application has been developed based on the theme laid out by the prototyping attempts
taken for the first report. Some of the functionalities displayed in the prototypes have been removed to respect
the timeline for the project. Changes have also been made to the interfaces. A large volume of academic
readings has been thoroughly read, both internal and external to the project, to apply effective changes to the
user interfaces that potentially increase the usability and aesthetics of the native application while strictly
following the design principles outlined in the readings. The description of the application focuses on the raffle
creator interface, raffle manager interface, ticket selling, ticket listing interface(s) and show winner interface.
The navigation aspects and additional functionalities have also been discussed in brief.
Raffle Creator Interface
The raffle creator interface implemented in our android application has some of the features from our prototypes
for project 1. The theme has remained the same, with the user entering data relevant to the raffle. Rather than
selecting a set of fields to be filled up later, the user will now need to enter information into the relevant fields to
create a raffle ready to go. A system timestamp has been utilized to fix a start date for the raffle.
Home Interface, Navigation and Additional Requirements
The home interface has been kept extremely simple as it has a minor purpose in terms of functionality as it can
be visualized as a door to functionalities. Extensive changes have been brought about to the navigation feature
as toolbars, and action bars with relevant icons and headings have been utilized over the button group from the
prototypes. In fulfilling the additional requirements, users can select a cover photo for their raffle in the creator
interface. They can select between margin or regular raffle from the raffle type selector. The margin raffle
functionality and OS sharing are yet to be implemented.
Raffle Manager Interface
The initial prototypes of the app did not include a raffle manager interface. The ticket manager interface has
been simplified and brought under the hierarchy of the new raffle manager interface. The functionalities of the
ticket manager prototype have been broken down to make them available through multiple interfaces.
Ticket
Selling
& Listing
Interface(s)
The ticket selling and interfaces have been developed as new interfaces over the previous all in one ticket
management interface introduced in the prototypes for project 1. Users can now click on a ticket list item to
make changes to the customer information of the ticket.
Show Winner Interface
The show winner interface shows the winning ticket number from the ticket list from a given raffle when the
Draw Random Winner button is clicked. The draw winner interface from the project one prototype can be found
to be a bit redundant as we already have a ticket list and all the necessary information. We need to pick a
random winner for a given raffle.
Testing Methodology
This section of the report discusses the user responses from the test users observed for each cycle of the think-
aloud test. The number of users involved in the testing process was 4, and the test was done on the
functionalities that were 'must have' according to project specifications.
Test User Details
➢ Test User Number 1: A Rana: Current ICT student with knowledge of Mobile App Design.
➢ Test User Number 2: T Apu: University colleague App UI designer for University Project Unit.
➢ Test User Number 3: M Islam: University colleague knowledgeable on user expectations.
The preparation, introductory materials and questionnaires were created for the testing cycles based on the
content from Week 6 lecture on usability think-aloud tests.
Preparation Tasks
✓ A video conferencing software was set up for all the participants and organizers to join.
✓ The video conferencing schedule, software setup and video participation tutorial videos were pre-recorded
and forwarded to the participants.
✓ The prototype interfaces for each prototype version were printed out for the test participants posted to them.
Introductory Material
The introductory materials were briefed to each test participant for each test cycle before the testing session was
initiated. The prototype screenshots were discussed sequentially with each test participant to gather information
as to what they assume to be the purpose of the application that would be followed by it. A set of tasks were
briefed to each user participant for a specific prototype version, based on their understandings of the prototype
versions. The users were guided as they interacted with the paper prototypes, and observation notes were taken
to identify usability issues.
Test Cycle 1: Project 1 Prototype 1 of Mirza Istiaque Ahmed
Tasks for Test User Number 1: Create a raffle and edit the raffle name
Steps were taken to complete the task: 12 actions
Total Completion time: 5 minutes and 21 seconds
Intervention attempts to guide the user when they failed to perceive action: 3 times
General observation notes on usability issues: The user took a top to bottom approach to interact with the
elements on the raffle creator interface. The two buttons at the bottom created confusion as to which action led
to the creation of the raffle. Users also faced difficulty editing the raffle name as selecting the right option and
making changes brought about issues relating to the fidelity of the interface components.
The task for Test User Number 1: Draw a winner for a given raffle
Steps were taken to complete the task: 6 actions
Total Completion time: 3 minutes and 50 seconds
Intervention attempts to guide the user when they failed to perceive action: 4 times
General observation notes on usability issues: The user ignored the generate button, which generated a list of
users registered for the raffle. Also, the user could not perceive the meaning behind the 'number of names' label.
User action lagged when selecting between Publish Button and Ticket Manager Button.
The task for Test User Number 1: Sell tickets for raffle and edit customer name for ticket
Steps were taken to complete the task: 8 actions
Total Completion time: 4 minutes and 23 seconds
Intervention attempts to guide the user when they failed to perceive action: 3 times
General observation notes on usability issues: The user faced difficulties perceiving the number of tickets that
could be sold for a given raffle. Also, the user did not receive any cues to confirm a ticket purchase to be
processed successfully.
Test Cycle 2: Project 1 Prototype 2 of Mirza Istiaque Ahmed
Tasks for Test User Number 2: Create a raffle and edit the raffle name
Steps were taken to complete the task: 10 actions
Total Completion time: 4 minutes and 10 seconds
Intervention attempts to guide the user when they failed to perceive action: 2 times
General observation notes on usability issues. The user had to come back to the editor interface even though he
looked to make the edits straightaway.
The task for Test User Number 2: Draw a winner for a given raffle
Steps were taken to complete the task: 4 actions
Total Completion time: 2 minutes and 56 seconds
Intervention attempts to guide the user when they failed to perceive action: 2 times
General observation notes on usability issues: The user ignored the attached name file, adding a list of users
stored in an external file. Also, the user could not perceive the meaning behind the 'number of names' label.
The task for Test User Number 2: Sell tickets for raffle and edit customer name for ticket
Steps were taken to complete the task: 8 actions
Total Completion time: 4 minutes and 20 seconds
Intervention attempts to guide the user when they failed to perceive action: 2 times
General observation notes on usability issues: The user could not perceive how many tickets he has selected for
sale.
Test Cycle 3: Project 1 Prototype 3 of Mirza Istiaque Ahmed
The task for Test User Number 3: Create a raffle and edit the raffle name
Steps were taken to complete the task: 5 actions
Total Completion Time: 3 minutes 20 seconds
Intervention attempts to guide the user when they failed to perceive action: 3 times
General observation notes on usability issues: The user tried to input data into the templates, which was a static
element inside the scroll view
The task for Test User Number 3: Draw a winner for a given raffle
Steps were taken to complete the task: 5 actions
Total Completion time: 4 minutes and 12 seconds
Intervention attempts to guide the user when they failed to perceive action: 1 time
General observation notes on usability issues: Users lagged when carrying out the repetitive task of entering the
names for the draw manually.
The task for Test User Number 3: Sell tickets for raffle and edit customer name for ticket
Steps were taken to complete the task: 8 actions
Total Completion time: 4 minutes and 41 seconds
Intervention attempts to guide the user when they failed to perceive action: 2 times
General observation notes on usability issues: The user tried to click on the disabled list button before selling
tickets for the raffle. No visual cue was provided as confirmation of the process, which caused further delay.
Questionnaire
▪ On a scale of 1 to 10, how easy was it for you to carry out the given tasks for each interface
❖ Test User 1: In terms of ease of use, 5/10 for creating raffle task 1, 7/10 for drawing winner and 7/10
for selling task.
❖ Test User 2: In terms of ease of use, 7/10 for creating raffle task 1, 6/10 for drawing winner and 6/10
for selling task.
❖ Test User 3: In terms of ease of use, 8/10 for creating raffle task 1, 4/10 for drawing winner and 6/10
for selling task.
Test Results
T1 – Create a Raffle and edit the raffle name
T2 – Draw a winner for a given raffle name
T3 – Sell tickets for raffle and edit customer name for ticket
U1 – Test User 1 with prototype 1
U2 – Test User 2 with prototype 2
U3 – Test User 3 with prototype 3
Completion Time results
In minutes
Task/User U1 for P1 U2 for P2 U3 for P3 Average Time Minimum Time
T1 5.4 4.2 3.3 4.3 3.3
T2 3.8 2.9 4.2 3.6 2.9
T3 4.4 4.3 4.7 4.5 4.3
Ease of use results
User ratings received for the task during the questionnaire. Rating out of 10.
Task/User U1 U2 U3 Average Rating
T1 5 7 8 6
T2 7 6 4 5.7
T3 7 6 6 6.3
Intervention Results
Intervention attempts when users lagged
Task/User U1 U2 U3 Average Interventions
(Rounded)
T1 3 2 3 3
T2 4 2 1 2
T3 3 2 2 2
Discussion
The native raffle draw application is an example of a Human-Computer Interaction model. In HCI, a goal is to
measure and improve the usability of software (Kluth, W, 2014). We have effectively measured qualitative
aspects of the application such a completion times, ease of use and number of interventions to effectively work
towards enhancing the usability for each interface developed to aid our users in carrying out a set of tasks. The
most common approach to evaluating mobile phone applications is the think-aloud method (Baharuddin et al.,
2013). Usability testing carried out via think-aloud led to the discovery of a large number of usability compared
to other methods such as heuristic evaluation and cognitive walkthroughs. The critical aspects of a usability test
include recognizing misuse, usability issues and the distinct non-predicted behaviours that help us understand
the mental models of individuals who will be the actual users of the application. A significant issue identified
during the usability test observation was that users were misusing UI elements (Fitts, 1992), e.g. the sell button
in the sell tickets task and the start button in the drawing a winner task. Users were also trying to navigate
between views, especially after the ticket selling task, which is identified as a minor usability problem (Ahmad
et al., 2006).
Focus on the usability testing results, the average ratings received in each category indicate that all three
prototypes share each other's usability strengths and weaknesses. In terms of the completion time, the average
time to complete task 3 is 4.5 minutes, close to the average time of 4.1 minutes for completing task 1 across all
three prototypes. From this, we can infer that both tasks carry similar mental loads and that bringing about
changes to one of them, such as centring elements and utilizing colours to make elements more visible to users
(Norman, 1988), will also reduce completion time. Task 2 received the lowest average score across the
prototypes in terms of ease of use, which means that the interface requires the most work and is likely to have
the highest number of usability issues relating to fidelity, perceiving functionality, and lazy labelling. The
average number of interventions required to help users complete the first task was three and the highest among
the other two tasks. We can infer that the users found themselves stuck when trying to create a raffle using the
raffle creator interface. The interface lacked details as to the task being accomplished and the right actions to
take to complete the task at hand. The average interventions decreased across prototypes 2 and 1 with an
unpleasant spike of 4 for Task 2 for User 1. In total, the two red flags on Task 2 and the relevant interface called
for the interface to be redesigned.
Conclusion
Finding an appropriate usability testing method to evaluate a mobile application has always been difficult
(Zhang and Adipat, 2005). On many occasions, software companies have strongly disagreed on letting end users
be a part of the software evaluation process, leading to its failure (Hussein, Mahmud, Md Tap & Jack, 2010).
The applications did not consider the mental models of their end-users. Hence, they were unable to understand
the features and components of the relevant application. Usability evaluation is a necessary process, and
choosing think-aloud testing is only logical given its success compared to other testing models when it comes to
evaluating mobile phone applications. However, our methodology and results section can be considered to be
limited in multiple avenues. E.g., most of the data collected were qualitative, which is not dependable when it
comes to identifying a problem pattern that can be solved at the root.
Furthermore, users were found to complete the task at hand in minimum time rather than perform the task
thoroughly, leading to users not following the instructions forwarded to them. An alternative approach
suggested utilising a virtual machine running screen capture software alongside the user video feed rather than
forwarding a set of prototypes or a website link (Huang, Bias, Payne, and Rogers 2009). Also, the System
Usability Scale questionnaire over a set of questions that aligns with the usability test cycles would have given
us a way to rank the contributions of each participant (Brooke, 1996). The mentioned usability test alternatives
and our test methodology limitation will be considered for further usability enhancement of the native raffle
draw application.
References
Ahmad, R., Li, Z., and Azam, F. (2006). Measuring navigational burden. In Software Engineering Research,
Management and Applications, 2006. Fourth International Conference on pages 307–314.
Baharuddin, R., Singh, D., and Razali, R. (2013). Usability Dimensions for Mobile Applications-A Review.
Research Journal Brooke, J. (1996) SUS: A "quick and dirty" usability scale. In
P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland (eds.) Usability Evaluation in industry.
London: Taylor and Francis. L of Applied Sciences, Engineering and Technology, 5(6):2225–2231.
Fitts, P. M. (1992). The Information Capacity of the Human Motor System in Controlling the Amplitude of
Movement. Journal of Experimental Psychology: General, 121(3):262–9.
Huang, S. C., Bias, R. G., Payne, T. L., & Rogers, J. B. (2009). Remote usability testing: A practice. JCDL
Proceedings 2009. 397.
Hussein, I., Mahmud, M., Md Tap, A. O. & Jack, L. (2010). Does user-centred design (UCD) matter?
Perspectives of Malaysian IT organizations. International Journal of Interactive Digital Media, 1 (1), 71-78
Kluth, W., Krempels, K.H. and Samsel, C., 2014, April. Automated Usability Testing for Mobile Applications.
In WEBSITE (2) (pp. 149-156).
Nelson, E.T. & Stavrou, A., 2011. Advantages and Disadvantages of Remote Asynchronous Usability Testing
Using Amazon Mechanical Turk. Proceedings of the Human Factors and Ergonomics Society Annual Meeting,
55(1), pp.1080–1084.
Norman, D. (1988). The Psychology of Everyday Things. Basic Books Inc.; Paperback Book Club edition
(1988).
Tullis, T., Fleischman, S., McNulty, M., Cianchette, C., & Bergel, M. (2002). An empirical comparison of the
lab and remote usability testing of websites. Usability Professionals Association Conference Proceedings.
Zhang, D. and Adipat, B. (2005). Challenges, Methodologies, and Issues in the Usability Testing of Mobile
Applications. International Journal of HumanComputer, pages 293–308.
