Query Signals Understanding TryRating

Guidelines FY25Q4

Overview
What is Query Understanding
What is the Intended App?
What are Genres?
What are keywords?
Rating Interface
How to Rate

● Query → Intended App: Relevant / Irrelevant

● Query → Genre 1 through Genre 6: Excellent / Good / Acceptable / Bad
● Query → Keyword 1 through Keyword 3: Excellent / Good / Acceptable / Bad

Additional Examples

● Example 1: query = “spotify”

● Example 2: query = “budget planner”
● Example 3: query = “workout app for home”
● Example 4: query = “languge lernin”

Overview
Welcome to the Human Rating Guidelines to rate the query understanding for the search terms that App Store users use to
search for an app that they are interested in.

Your role is to identify the relevance of the intended app, genre, and keywords associated with the search query.

In total, there are a maximum of 10 Relevance Ratings for each query, and these ratings are distributed in the following way:

1. <Query, Intended App> Relevance Rating

2. <Query, Genre> Relevance Rating (Maximum 6 Genres)
3. <Query, Keyword> Relevance Rating (Maximum 3 Keywords)

What is Query Understanding

User queries are the strongest indicator of user intent in App Store search, thus understanding query is critical to search ads
success. Meanwhile, the user queries can be challenging to understand since queries can be noisy, ambiguous, abbreviated,
multilingual, misspelling etc.

Query understanding refers to the process where a system interprets the intent of a user’s query to deliver more accurate and
relevant search results. In this project, we provide 3 types of enrichment information for each query to help interpret the given
query. You are expected to rate if these enrichment tags correctly captured the query intent

● intended_app: the app entity explicitly searched in this query.

● intended_microgenre: A granular category requested by this query. The value uses predefined taxonomy with ~529
values.
● related_keywords: any keywords that’s semantically similar or relevant to this query.

In the context of this rating task, you are expected to assess how well the system has understood the query — whether the
outputted tags has correctly represented the user’s intent.

What is the Intended App?

The intended_app tag will be the entity that’s explicitly asked for in the query. You will see this value is blank if QU system
thinks no app is explicitly asked.

The exact string name might be different, but the guiding principle is “as long as the entity in intended_app and the
original query refers to the same entity, it should be rated as relevant. The lexical spellings doesn’t matter ”

Examples:

Query intended_app rating Note

spotify spotify Intended
spotify app spotify Intended
The app entity in query and intended_app are
spotify apple music Unintended two different apps
music player spotify Unintended no app is explicitly asked in the query
Even the string value are not exactly the
uber uber - request a ride Intended same, as long as both are talking about the
same, it should be relevant

chikc dil a chick fil a app Intended

regardless of typos and different phrasings,
chick fi la app chick fil a app Intended the entity in query and intended_app are the
same

chickfila 1 app chick fil a app Intended

The query is not sepcifically asking for chick fil

chicken sald chick chick fil a app Unintended a
ig Instagram Intended abbreviations are fine
インスタグラム Instagram Intended different languages are fine

coffee starbucks Unintended the query is not explicitly asking for starbucks

New York City official

transit app MYmta Intended

What are Genres?

Genre definition : Genre is a more granular category info, which captures the underlying features, functions, or content types
requested by the user. The value of the genre is from a fixed taxonomy.
 ● Genre Description: genre will come with genre description that defines what genre means. Please read the description
and decide if the query intent is relevant to this genre
● Special Note: Soemtimes the Genre Description is missing, please rate based on the genre only.

Examples:

Query genres rating Note

forest Health and Excellent the query is asking for an app forest, whcih is
Wellness a wellness app

feeld Dating and Excellent the query is asking for an app feeld, whcih is a
Romance dating app

toyblast Casual Excellent

inoreader news rss News Aggregator Excellent

reader
getting out We need to intepret query in app store
outdoor bad context. There is an app called "getting out",
which is a prisoner messaging app
Look up this query in app store or google
安逸花 Gardening bad search, this is the name of a load service app

安逸花 Loan Services Excellent

taobao the descirption of "general service" is apps

general service bad providign freelancer services. Taobao does
not qualify
Alltrails Camping good
grocery shopping Coupons and
Discounts good

What are keywords?

● Keywords
Any freeform keyword that is semantically related to the query that does not fall in the genre’s taxonomy. The keywords
can by abbreviations, synonyms, related functions etc, basically anything related

Examples:

Query keywords rating Note

zoom video meeting Excellent
zoom business meetings
Excellent

飞猪 US tour
acceptable
the query is referring to an travel booking app,
not strictly related to US tour
飞猪 online travel agency
Excellent

zoom
shopping bad zoom doesn't have any shopping functions

online games chess chess is a offline game, but is accepatble as

acceptable chess is still a game
Rating Interface
You can see the rating interface in the below screenshot.

● On the left-hand side of the rating tool interface, you will see the user search query.
● Next to the search query, you will see the links you can use to help begin your research on the query and what it means.
Please do the research on what this query means in the App world first.
● Under these links is another link to the App Store search results for the query. This will show you the results a search for
the query at the App Store would give. Remember to look at the search results only that appear at the top of the result
page. Use these links to research the query only.

Other things that appear on the left side are as follows:

● intended App: This is QU output value for intended app.

○ Note: If no Intended App appears, you will not need to rate anything here.
● Genre 1 to 6: This is QU output value for genre. Each query can have 0-6 genres. You will see Genre description below
the genre name, which is describes what each genre mean. Please read before rating.
○ Genre Description: You will find this right below the genre. This is meant to give more information about the
genre.
○ Note: If the Genre Description is missing, please rate it based on genre
● Keyword 1 to 3: This is QU output value for keywords. Each query can have 0-3 keywords

Please note that the only research options you have are regarding the search query.

Use your research, intuition, experience, and knowledge to rate the relevance of queries with intended app, genres, and
keywords.

** Important Note **
Please note that since this template is dynamic in nature, if an entry such as Intended App or Genre 1-6, or Keyword 1-3 is
empty or null, it will not be shown on the rating interface. Therefore, no two queries will have same rating interface as it will
depend on whether that query has associated intended app, genre 1-6, or keyword 1-3 in the input source data or not.

How to Rate
In this section, you will learn how to assess the relevance of query understanding outputs by rating:

● The relationship between the user query and intended app, if shown, (binary: relevant or irrelevant)
● The alignment of the query with each predicted genre (one of Excellent, Good, Acceptable, Bad)
○ Special Note: If the Genre Description is missing, please rate it based on genre name
● The alignment of the query with each predicted keyword (one of Excellent, Good, Acceptable, Bad)

These ratings help us evaluate the quality of our query understanding system, which predicts an "intended app", a list of
genres, and a list of keywords for every search query.

Please pay attention to the following while performing these rating tasks:

● Don't rely solely on keywords in the query — infer the user's true intent.
● If unsure, use your best judgement to provide the appropriate rating and level of relevance.
● For genres or keywords with null or missing entries, please do not rate them as they are placeholders with no
value

Query → Intended App: Relevant / Irrelevant

GOAL

Determine whether the predicted intended app is the same app user is looking for in the query. It doesn't matter whether the
intended app actually exists on the app store or not.

If the query is in foreign language or a language that you don’t understand, you must still rate the task. Please use the
provided research links in Web Search for Query section to understand what the query means and the intent behind it.

RATE AS RELEVANT IF:

● Principle: rate relevant if query and intended_app are the same app entity, regardless of syntactic spelling.

RATE AS IRRELEVANT IF:

● The app is not what user asked for in the query explicitly

EXAMPLES
Query Intended App Rating Explanation

"spotify" Spotify Relevant This is rated Relevant because intended app is a direct match to
the query and the user's intention is to find an app.

This is rated Irrelevant because intended app is not a direct

match to the query and the user's intention is to find an app with
"fitness tracker" Fitbod Irrelevant the name fitness tracker which is different than fitbod. Fitness
tracker query indicates that the user is interested in tracking
something related to their fitness, whereas fitbod doesn't directly
match that expectation, as previous example of spotify.

This is rated Irrelevant because user is not explicitly asking for

"free calculator" TikTok Irrelevant this app with this query.
This is rated Relevant because intended app is a direct match to
"uber" Uber Relevant the query.
Divide and This is rated Irrelevant because intended app doesn't serve
"surrogacy" Conquer Irrelevant anything closely related to the query.
"yutube" Youtube Relevant This is rated Relevant because even though query has a typo, i
t's clear user wants to search for Youtube app

‫واتساب‬ WhatsApp Relevant This is rated Relevant because the user is interested in
Messenger WhatsApp as the English translation of the query is WhatsApp

whatsapp WhatsApp Relevant This is rated Relevant because the user queried for WhatsApp
Messenger and got the result as an app called WhatsApp Messenger

chickem salad Chick-fil-a Irrelevant The user query didn't intend for an app specifically
chick

chickfila deliver Chick-fil-a Relevant The user query clearly wants chick fil a app

chick fil a ap Chick-fil-a Relevant The user query clearly wants chick fil a app

New York City

official transit app MYmta Relevance

京东 jd.com Relevant the are the same entity

Query → Genre 1 through Genre 6: Excellent / Good / Acceptable / Bad

GOAL

Evaluate how well each Genre 1 to Genre 6 captures the user's intent behind the query.
Each query may be associated with up to 6 genres. You will rate each genre individually.

RATE AS EXCELLENT IF:

● Genre is a direct match or ideal representation of what the user meant.

● Genre reflects the core feature or function the user is looking for.

RATE AS GOOD IF:

● Genre is clearly relevant but not the most precise.

● Still likely useful or interesting to the user based on the query.
RATE AS ACCEPTABLE IF:

● Genre is loosely connected to the query.

● User would not be surprised to see it, but may not find it especially helpful.

RATE AS BAD IF:

● Genre has little or no logical connection to the query.

● User would find it confusing, misleading, or completely off-topic.
● The genre is out of taxonomy, ie it has no genre description

EXAMPLES

Query Genre Rating Explanation

"photo editing" Photography Excellent This is rated excellent because Genre is direct match to the
query

This is rated good because the Genre is clearly relevant to the

query and can be interesting for the user. When the user is
"weight loss" Meditation Good searching for a weight loss, they might be interested in tips and
ways to get rid of their weight, and meditation is one of the ways
to do that. Therefore, it is rated Good

Online This is rated acceptable because the Genre is loosely related to

"facebook" Marketplace Acceptable the query. Online Marketplace is one of the features of the app
suggested by the query.
This is rated bad because there is no connection between the
"budget app" Gaming Bad genre and query.

Query → Keyword 1 through Keyword 3: Excellent / Good / Acceptable / Bad

GOAL

Judge how well the predicted keywords (up to 3) reflect specific features, intents, or entities implied by the query.
Each keyword should provide additional useful context not captured by the main genre(s).

RATE AS EXCELLENT IF:

● Keyword captures a critical element of the query.

● User is very likely expecting this concept or feature.

RATE AS GOOD IF:

● Keyword adds helpful, specific context that aligns with the query.

RATE AS ACCEPTABLE IF:

● Keyword has some weak relevance to the query but is not very helpful.
● Its presence is reasonable but not compelling.

RATE AS BAD IF:

● Keyword is unrelated or misleading.

● It might suggest incorrect features, apps, or domains.
EXAMPLES

Query Keyword Rating Explanation

"budget planner" "expenses" Excellent This is rated excellent because the keyword is an essential part
of the user search query and the intention behind it.

This is rated good because the user intrested in a music

"music download download app, which is not a direct match to streaming, but, it
app" "streaming" Good can be one of the additional features that the user might find
interesting.
This is rated acceptable because even though it is not a direct
"online games" "board games" Acceptable match, some people might find the relationship between query
and keyword as relevant.

"baby tracker" "stocks" Bad This is rated bad because there is no match whatsoever
between the query and the keyword.

Additional Examples
This section contains examples of some queries, intended apps, genres, and keywords, and their respective relevance ratings
along with the reasoning for that rating.

Example 1: query = “spotify”

QUERY → INTENDED APP

● Spotify: Relevant because Spotify is the exact match and fulfills the user's likely intent.
● Fitbit: Irrelevant because Fitbit is for fitness tracking and has no music-related purpose.

QUERY → GENRE (GENRE 1 THROUGH 6)

● Music Streaming: Excellent because it is core functionality of Spotify and it perfectly aligns with query.
● Podcast Player: Good because Spotify (query) includes podcasts as a feature.
● Video Player: Acceptable because Spotify (query) includes limited video content and is not the main feature.
● Fitness Tracker: Bad because it doesn’t have any relationship to Spotify (query)

QUERY → KEYWORD (KEYWORD 1 THROUGH 3)

● playlists: Excellent because it is central to Spotify User Interface.

● audiobooks: Good because this is one of the features in Spotify.
● radio: Acceptable because it has some similarity with Spotify (query) but is not the main focus.
● step counter: Bad because the keyword doesn’t bear any relationship to Spotify (query)

Example 2: query = “budget planner”

QUERY → INTENDED APP

● Budget Planner App: Irrelevant because the query is functional, and is not explicitly asking for a query.
● Candy Crush: Irrelevant because it is a game with no financial planning features.

QUERY → GENRE (GENRE 1 THROUGH 6)

 ● Personal Finances: Excellent because genre directly addresses user intent.
● Expense Tracker: Good because this genre is closely related and useful to the query.
● Spreadsheets Tools: Acceptable because this genre might be related to budgeting but not designed specifically for it.
● Games: Bad because this genre is unrelated to the query.

QUERY → KEYWORD (KEYWORD 1 THROUGH 3)

● budgeting: Excellent because this keyword perfectly aligns with the query’s main theme.
● saving goals: Good because this keyword represents a common functionality supported by the app suggested for the
query.
● reports: Acceptable because this keyword could be useful for users in the context of the query.
● gems: Bad because this keyword doesn’t have anything common with the query.

Example 3: query = “workout app for home”

QUERY → INTENDED APP

● workout app for home: Irrelevant because the query is functional, and is not explicitly asking for a query.
● Uber Eats: Irrelevant because intended app serves entirely different purpose than the query.

QUERY → GENRE (GENRE 1 THROUGH 6)

● Home Fitness: Excellent because it is a direct match to the query.

● Health & Wellness: Good because it comes under a broader category, but, still relevant to the query.
● Meal Planning: Acceptable because this genre might have a tangential connection the query and may serve indirect
query intent.
● Food Delivery: Bad because it is a direct contradiction to the query.

QUERY → KEYWORD (KEYWORD 1 THROUGH 3)

● bodyweight exercises: Excellent because this keyword is in extreme alignment with the query.
● no equipment workout: Good because it supports query’s constraints.
● calories tracker: Acceptable because it is related to the user query, but is not the core focus of the query.
● delivery trip: Bad because it doesn’t belong to the same domain as the query and is unrelated.

Example 4: query = “languge lernin”

QUERY → INTENDED APP

● Language Learning: Irrelevant because the query is functional, and is not explicitly asking for a query.
● Google Maps: Irrelevant because there is no direct connection between the query and the intended app.

QUERY → GENRE (GENRE 1 THROUGH 6)

● Language Education: Excellent because it directly aligns with the query’s intent.
 ● Flashcard Learning: Good because it implies the functionality that the user is looking for through the query.
● Travel Tools: Acceptable because the query can be used to relate to this genre for a small portion of the population.
● Navigation: Bad because there is no direct or indirect relationship between this genre and the query.

QUERY → KEYWORD (KEYWORD 1 THROUGH 3)

● vocabulary: Excellent because it is a central concept in the query.

● grammar: Excellent because it is one of the features the user look for when they search for the given query.
● culture: Acceptable because it is indirectly related to the query. It may enhance the user experience with this keyword,
but, is not essential.
● traffic: Bad because this keyword is unrelated to the query.