Mobile App and Query Evaluation for Google Searches
Version: 2016-11-08
In this task, you will research a mobile app and a query entered in Google Search. You will then
answer a question about how well the app fulfills what the user is looking for.
Terms
● You already know what a query is: it is a word or phrase that a real person typed into a
Google search.
● A mobile device is a small, handheld computing device, such as a smartphone or tablet.
● A mobile app is a smartphone or tablet application.
How to rate this task
Step 1: Review and understand the user query.
You will see a query that a user entered. Just as in normal search ads rating, it will be your
responsibility to do your best to understand what the user was looking for. Y ou will be asked to
assume that the user entered the query in a regular search on Google.com on a mobile
device. Although this evaluation is concerning the relevance of mobile apps, do not assume
that if a user entered a search into Google that they are searching for an app. Instead, consider
the user to be an average person entering the query into Google search. For more information
on how this is applicable to the task at hand, please see examples in the table below Step 3.
Unrateable Query
If the query is in a foreign language, is incomprehensible, or otherwise can’t be rated, select the
“Unrateable query” flag and submit the task.
Step 2: Familiarize yourself with the mobile app.
Click the button labeled “research mobile app” to open the app’s Google Play page, then review
its content. You should consider all the information available to you in order to inform your
understanding of the app, including the text that describes the app, ratings, any relevant
comments from users, and even your overall impression of how professional and trustworthy the
app looks. Some additional notes:
● Do not download the app.
● The Play Store page may include some additional content, such as links to other parts of
the Play Store, or lists of “Similar” mobile apps. Please ignore this and only focus on the
content that is related to the main app on the page.
● You may, however, follow links and do additional research outside of the Play Store if it
helps you understand the app.
Unrateable Apps
If the Play Store page or the app it describes is in a foreign language, is incomprehensible, or
otherwise can’t be rated, select the “Unrateable app” flag and submit the task.
Personal Knowledge of Apps
If you've personally installed and used the app, or believe you have additional knowledge about
the app that would affect your ratings in this task, select the flag labeled "I am personally familiar
with this app." You will still be required to complete the task.
Step 3: Answer the following question: Would installing and using the app satisfy the
user intent?
Think about what the mobile app is used for and what the user was looking for. For example, if
the app’s purpose is to deliver food and the provided query contains the word “fruit ninja”, then
the user intent is unlikely to be satisfied by the user installing and using the app. Pay attention to
the app’s quality as well. A good quality app looks trustworthy, has a well-written description that
makes the app’s purpose clear, and gets positive reviews from users. The user intent is less
likely to be satisfied by a poor quality app.
Taking all of these factors into consideration, select a rating from [-100] to [100] to indicate the
likelihood that a specific user would be satisfied by installing and using the app.
Rating Description Examples
Satisfaction Likely The user is looking for an app, and User query: [flashlight app]
this one provides exactly the App: S
uper-Bright LED Flashlight
functionality that the user is looking The user can turn their phone into a
for. The app seems to be of high flashlight with this app.
quality (i.e., it's not scammy).
If the user is looking for a specific
app but is presented with a plausible
substitute, you should select this User query: [caviar food delivery]
option. App: D
oorDash
While the user is looking for a
specific food delivery app - Caviar -
the DoorDash food delivery app is a
plausible substitute.
Satisfaction Possible The user might install and use the User query: [flashlight]
app because the query has multiple App: S uper-Bright LED Flashlight
equally likely user intents, namely The user is equally likely looking for
the user could be looking for a a flashlight app as s/he is looking for
physical good or app. The app can a physical flashlight for sale.
only satisfy one of the plausible
intents.
User query: [airline]
App: K ayak
There is no app that could possibly
receive "Satisfaction Likely" for the
query [alrline] - it is simply too vague
of a query. However, a subset of
users might find Kayak, a flight deal
app, useful. (Note: other airline apps
like Southwest or Delta would also
be Satisfaction Possible).
User query: [facebook]
App: Facebook
The user may be looking for the
Facebook app, but it's also likely that
the just want to log in through
Facebook's website as quickly as
possible.
User query: [install facebook]
App: S napchat
The user is looking for a specific
app. In some ways, Snapchat is
similar to Facebook, but it's not close
enough to be a true plausible
substitute.
Dissatisfaction The user probably won't install the User query: [board games]
Possible app because the user intent is App: C heckers Elite
unlikely to be met by the app, The user intent is much more likely
and/or because the app appears to to find physical board games rather
be of low quality. than the app-related version, but it is
still possible that the user might
download the app.
User query: [veterinarian game app]
App: V
et Clinic
The user likely won’t install this app
because it is very low quality: it has
bad reviews in the Play store and the
app description is riddled with
spelling and grammatical errors.
Dissatisfaction Likely The user won’t install the app User query: [bicycle]
because the app is unrelated to App: B ike Mayhem Free
what they are looking for, because If a user enters [bicycle] into a
the app is of obviously low quality, Google search query, it is highly
or because the query has multiple unlikely that the user is searching for
possible user intents and the query an app.
is unlikely referring to an app.
