10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine

Search - Apple Music Training

Guidelines for Search - Apple Music Training

Updated:
March 2022: Additional guidance on queries for Vocaloid artists.
May 2022: Exception added for genre queries looking for noises, not music.
June 2022: Additional examples on Soundtrack queries.
Sep 2022: Promoted/related content of Acceptable song is Off-Topic. New ratings for Playlists results for Song Nav.
Oct 21st 2022: Removed "Unacceptable" rating for playlists with 200+ songs. Added rules for regional charts ratings in Genre section.
Oct 24th 2022: Removed query type screenshots and added clarification to Song and Artist for playlist ratings. Essentials for Song Nav –> Good.
Dec 1st 2022: Additional visual guide on Song Navigational.
Dec 21st 2022: Additional notes on Popularity and Recency, guidance for Video Navigational and Classical Music queries.
Feb 28th 2023: Songs by the artist from compilations are rated Good instead of Excellent.
Apr 7th 2023: Intended song from compilation album is rated Excellent instead of Perfect, the album/playlist with such song –> Acceptable
Sep 6th 2023: Added more explanation on matching Artist names for secondary intent under Artist Nav Acceptable/Excellent
Sep 21st 2023: Album "Single" with more than one song are to be considered as albums, not songs

Music Search v3 Guidelines

Introduction
In this document, we explain relevance rating guidelines for Apple Music Search. You will use the BaseLine tool to make these ratings.

What is Apple Music?

Apple Music is a music and video streaming service developed by Apple. Users select music to stream to their device on-demand, or they can listen to
existing, curated playlists. The service also includes the Internet radio station Apple Music 1 (previously known as Beats 1), which broadcasts live to
over 100 countries 24 hours a day. Some highlights of what Apple Music offers to the users :
Stream 100 million songs ad‑free.
Download your favorite tracks. Play them offline.
Get exclusive and original content.
Listen across all of your devices.
Sing along, tap ahead, or just listen with lyrics view.
Listen live to local radio stations from around the world.
Discover songs you’ll love from music picked just for you.
Tap into new music with curated playlists from our editors.

The importance of your work as a Rater

The data we receive from you in the form of high quality relevance judgments will be used to build and improve artificial intelligence systems such as
search algorithms and machine learned rankers that power the user experience for Apple Music users.
Our ultimate goal is to surprise and delight our customers by improving search quality and enhancing customer satisfaction, and you play an important
role in this.

Ask yourself “Why is this particular result returned for this query? Is this relevant?” Stay curious and do thorough research to understand the
relationship between a query and the result.
Your attention to detail, research and language skills as well as your cultural knowledge of the market are all critical to the success of our projects.
Please keep in mind that your tasks will be spot-checked for quality, and measured against those of your peers. If your accuracy rate is consistently
high enough, you may be qualified to work as an auditor who gives feedback to your peers.

Query Research
Query Intent
We expect you to use search resources such as Bing, Google, Yahoo, YouTube, Genius as well as Apple Music to help you understand the intent of
each query. Searching "[query] apple music" or "[query] iTunes" will help narrow down the intent. More on User's Intent in the Results Research
section below.

Primary and Secondary Intent

The primary intent of a query is the most likely intent, i.e., the intent of most users who enter that query in Apple Music.
A secondary intent, on the other hand, is less likely, or would be a less popular intent compared to a primary one. A secondary intent could be:
content relevant to a smaller group of users than for the primary intent
lower quality/lesser known content such as unpopular covers, remixes from unknown artists.

Query Type
Each query is assigned a ‘Query type’. On the left hand side in BaseLine, inside the input metadata section you will find the classification of the query.
The classification is already set (you do not have to classify the query yourself).
You should rate the content relevancy based on (1) the classification of the query and (2) how relevant is the content in relation to the query and the
query type.

The ‘Query type’ aims to help you determine the primary intent of the query. Please also consider potential secondary intents when doing your ratings.
Should you disagree with the ‘Query type’ classification, please explain in the comment section why it should be different. You can then apply the
appropriate rating scale to complete your judgement.
Example Comment: “I believe that [red] shall be classified as Album rather than Soundtrack. I used this query type for the rating.”

Misspelled Queries
Some queries are misspelled by our users. When this is the case, assume the query intent is for the correctly spelled query.
If the query is [the weekend] or [lasy gaga], the user likely means [the weeknd] and [lady gaga]. Use your best judgement.
Missing words like feat., x, ft. in Artist Queries (like [nicki minaj lil baby] instead of [nicki minaj ft. lil baby]) would not be considered spelling errors, as
there is nothing that's misspelled.

Results Research
The results are generally rated based on the query type (section: Relevance Ratings based on Query Type). However, there are three general aspects
to consider when rating any result: user's intent, popularity and recency.

User's Intent
You will come across more ambiguous queries with multiple intents (like [adore you]) or queries with a very narrow specified intent (like [adore you
jessie]). For queries with unclear primary intent, popularity and recency of returned content plays a bigger role. For queries with narrow clear intent,
only the relevance of the result plays a role in ratings. See examples of two search queries and their ratings below.

Example 1: [adore you] –> The user’s intent is not clear/specific, since there are multiple songs with the title [adore you]. Therefore, we base the
ratings of the results on popularity to satisfy the intent for the majority of users.

https://baseline.apple.com/training/evaluations/4/guidelines 1/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
The song ‘Adore You’ by Harry Styles > Perfect
The song ‘Adore You’ by Miley Cyrus > Excellent
Harry Styles’ Artist Page > Good
The song ‘Adore You’ by Jessie Ware > Acceptable
The album ‘What’s your pleasure?’ by Jessie Ware (containing the song ‘Adore You’) > Off Topic
Jessie Ware’s Artist Page > Off Topic
Example 2: [adore you jessie] –> This query specifies the artist along with the song name, so we are now sure of the intent. Therefore, the ratings are
based only on relevancy to the query, disregarding popularity.
The song ‘Adore You’ by Jessie Ware > Perfect
The album ‘What’s your pleasure?’ by Jessie Ware (containing the song ‘Adore You’) > Good
Jessie Ware’s Artist Page > Good
Other music by Jessie Ware > Good
HAAi’s Remix of ‘Adore You’ by Jessie Ware > Acceptable
The song ‘Adore You’ by Harry Styles > Off Topic
The song ‘Adore You’ by Miley Cyrus > Off Topic
Harry Styles’ Artist Page > Off Topic

Popularity
If the user's intent is clear and narrow (like [adore you jessie] in the section above), popularity and recency doesn't play a role in ratings.
For queries with unclear intents, popularity of results helps us understand how many users we would satisfy with the following result. In other words,
more popular content would be relevant to a larger group of users, so the result is more likely to satisfy ambiguous intent.
There are multiple resources you can check to understand popularity of content in your storefront: Bing, Google, Yahoo, YouTube, Genius, Apple
Music. Please make sure you use at least two resources before making your decision.

Google
You can use Google to research the popularity of returned content. Searching "[query] apple music" or "[query] iTunes" will help narrow down the
intent.

YouTube
1. Total Views on YouTube
To assess popularity of the result, you can look at the number of views for the song/artist (make sure to sum the number of views for the song or artist
if there are multiple relevant videos for that content).

2. Average Monthly Views Since Upload (AMVSU)

This metric helps us look at popularity of content, taking into account recency as well.
If there are 2 popular songs with matching title and relatively similar number of average views, both songs can be eligible for Excellent. The metric we
adopt is Average Monthly Views Since Upload (AMVSU) = Total Number of Views ÷ Months since release. This metric can also help you decide
between primary intent (Perfect) and secondary intent (Good). See example below.
Example: [freedom] –> The query has a slightly ambiguous intent that can refer to Pharrell Williams' or Jon Batiste‘s songs.
Pharrell William's song has 127M views on YouTube over the last 7 years. AMVSU –> 127÷84 = 1.5M views per month since upload
Jon Batiste‘s song has 14M views over the last year. AMVSU –> 14÷12 = 1.2M views per month since upload.
Therefore, even though the Pharrell William's song has more views overall, Jon Batiste‘s song can also qualify for the rating Perfect, given its recency
and monthly popularity.

3. Artist's Top Songs

When researching popularity of the artist, you can type the name in search to check YouTube's artist page with top songs and albums listed below the
name. Pay attention to average number of listens/views for a song to understand artist's overall popularity.

Spotify
Spotify can be a good source for assessing popularity of content. You can use metrics such as:
1. Number of monthly listeners
2. Number of artist's followers
3. Plays on individual songs
4. Verified artists (with blue checkmark) is also a good indicator of artist's popularity.

Apple Music
You can use Apple Music's search results, top charts playlists and the page Top Songs by the artist as a supporting popularity data point as well. Make
sure to use other resources in your research as well.

Music/Search platforms relevant in your storefront

Popularity of content should be assess world-wide and for the specific storefront you're working with. If the content is not popular worldwide but is very
popular in your storefront, the result will have higher rating due to high popularity.

Recency
There are cases were a song was released in the last couple of months and might not have as many views as another older song with similar title.
There is a possibility that the more recent song is the intent as it is currently charting and/or trending. For cases like this, both the recent song and the
older song with more views can both get a rating of Perfect/Excellent.
Example: [Let It Go]
Let It Go from the movie Frozen is the most popular song with matching title with 2.5 billion views. This Song would get a rating of Perfect, since it
would be considered primary intent.
The song Let It Go by DJ Khaled ft. Justin Bieber and 21 Savage is less popular with 20 million views. However, the song is from a little over 1
month ago and was in the top charts. This is a possible primary intent and therefore is eligible for Excellent along with Frozen song (as of 2021).
The song Let It Go by James Bay is another popular song (368 million views) with matching title. However, this song is from 6 years ago, and is less
popular than the song from Frozen. We cannot use recency in this case. The best rating for this song is Good because it satisfies secondary intent.

Tips & tricks

Preview pages
Each search result will have a link to an Apple Music Preview page. This preview page will provide you with more details about the result, and allow
you to listen to snippets of songs. For Radio results, please judge based on relevance. For example, if your query is [hip hop], and the returned
result is Hip-Hop Radio by Apple Music, it's clear that the result is highly relevant, and there's no need for an Apple Music Subscription to listen to the
station to verify relevance. Should the query intent be for a country song, and the returned result is a Hip-Hop Radio, you can rate that result as Off
Topic. Should a Country Radio station be returned for a country song query, that radio can be rated as Good (again, without listening to the radio, just
based on the genre of the song and the station).

Singles (Album Container)

Sometimes you will encounter song singles in album container. Here's an example of a Taylor Swift's song Ronan in an Album Single container. This
can be considered equivalent to a song output and rated accordingly.
EP could be treated as albums if they contain multiple songs. If the result has "– Single" in the title but contains more than one song in the container
(like an example of this album – Single with two songs in it) it should be treated as album.

https://baseline.apple.com/training/evaluations/4/guidelines 2/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine

Rating Non-English Queries

Please note that Apple Music users query for content in multiple languages, not only in English.
All queries in all languages are valid search queries, and the results should be rated based on relevance and quality, just like with any other type of
music query. Utilize translation tools and conduct thorough side searches to understand the likely user intent.

Rating New Releases

When rating newly released content, you might see that the online popularity of the content (YouTube views, media, Google search results) is still low.
Take into consideration the potential popularity of the item (based on artist popularity, anticipation of the release, historical popularity, etc) and apply
the rating accordingly. For example, a newly released content by a popular artist can be considered secondary intent and not be counted as
"unpopular matching content".

Seasonal Results
In some locales there are seasonal results which have additional relevance and popularity at specific times of year, for example Christmas songs are
very popular in the US at the end of the year. Please consider the seasonal popularity when rating results and consider increasing the rating due to
seasonal popularity. E.g. Jazz Christmas playlist should be rated higher for a query [jazz] around Christmas time.

Classical Music Queries

Sometimes you can come across a query with a classical music intent (composer, composition name, playlist, etc).
For queries related to classical music, Composers are classified as Artists. All works composed by the queries composer can be treated as
content by the artist. If not listed in the container itself, please research the work title or album title to check whether the content is by the right
composer.
Classical Work Name or Catalogue Number can be treated as Songs. Sometimes a classical composition can consistent of multiple tracks
(movements). The composition can be aggregated in a Song, Album or Playlist container. Work/Compositions: albums/playlists/song containers that
consist the full intended Work or only part (one movement) are considered Perfect. Please note that classical music composition can have a
numeric name (like “Piano Concerto No.14, Op.27) or AKA name (like “moonlight sonata”), but both point to the same piece of work. When coming
across classical music query, please research the relevant composer and composition online to asses relevance of the results.
Relevant sections (Song Navigational and Artist Navigational) provide additional examples and support for those.

Problem: Other
Use this rating when a problem or technical issue with the task in BaseLine that makes it impossible to judge relevance. For example a piece of
content with an unavailable title, artist, and thumbnail image.

image unavailable.png •6.42 KB

Problem: Other rating should not be used if the content is unavailable in Apple Music but it is still possible to determine relevance between input and
output based on the result title or artist name and by completing a side search on Google and/or on other streaming platforms such as Spotify, Deezer,
etc.

Unacceptable: Other
Please disregard the rating Unacceptable: Other and only use Unacceptable: Off-Topic for Unacceptable results.

Mandatory Comment
Each rating must be explained in the comment box. Even if “optional” is indicated, rating comments are always mandatory.
The comment should be concise and must only explain why the rating chosen is the correct one in application of the guidelines.

Relevance Ratings based on Query Type

In relevance rating, we start with a query and score the relevance of content for that query. Let’s look at different query intents and some rating
examples.

Artist Navigational
Perfect: a piece of content that clearly satisfies the intent of the query with clear intent.
The artist page for the band/musician/composer.

1. [lady gaga] – the artist page for Lady Gaga on Apple Music
2. [bruno mars] – the artist page for Bruno Mars
3. [bealtes] – the artist page for the band, The Beatles. (Note that the query is misspelled.)
4. [billie eilish] – the artist page for Billie Eilish
5. [mozart] – artist page for Mozart

Excellent: content by the primary intent artist (in different music containers).
Songs, albums, and music videos uploaded by the artist, including featuring artists (regardless of popularity).
Important: The song needs to come from the artist's uploaded container (e.g. official album) or an official album of the feature song to be
qualified for Excellent. Songs by the artist from compilation Various Artists albums are rated Good. Soundtracks are not counted as Various Artists
compilations and can be considered Excellent.
Note that covers/remixes of the artist's content are demoted to Acceptable, unless the query's intent is the DJ and the result is a remix by that DJ
(in that case, it's considered content by the intended artist). For more info, refer to examples 11 and 12.
Note that to qualify for Excellent the content needs to be by the primary intent artist, not simply match the query. For example, for the query
[pink] the content by unpopular artist Pink would be Acceptable instead of Excellent, since the intent here is artist P!nk and only the content by
that artist should be considered Excellent.
Playlists featuring only the intended artist (Essentials, Deep Cuts, Next Steps).
Radio shows & episodes hosted by the queried for artist.
Songs that the artist is featured on.
Additionally for classical music queries:
Individual works by the composer (regardless of popularity or the artist performing). Works can include multiple tracks and can be in aggregated in
songs/albums/playlists containers.
Composer station by Apple Music Classical (for composer Artist queries)
Playlists or Various Artists albums featuring only the music by composer (performed by other artists) .

1. [lady gaga] – Lady Gaga's album The Fame Monster

2. [sean paul] – song Baby Boy (feat. Sean Paul) by Beyoncé (song from the official album of another artist with the featured song)
3. [idina menzel] – song Let It Go from an official soundtrack album
4. [ed sheeran] – song Perfect from the artist's official album ÷ (Deluxe)
5. [bruno mars] – a music video featuring Bruno Mars
6. [taylor swift] – playlist by Apple Music Taylor Swift Essentials
7. [eminem] – the song Believe by Eminem
8. [nicky minaj] – Queen Radio hosted by Nicky Minaj
9. [billie eilish] – song Lovely by Billie Eilish and Khalid
10. [Doja Cat] – song I Like You (A Happier Song) [feat. Doja Cat] by Post Malone
11. [Ed Sheeran] – song For My Hand (feat. Ed Sheeran) [Mixed] (a remix by DJ Neptune of Ed Sheeran’s song) –> Acceptable
12. [Meduza] – song Bad Habits (MEDUZA Remix) [Mixed] (a remix by MEDUZA of Ed Sheeran’s song) –> Excellent
13. [mozart] – The 99 Most Essential Mozart Masterpieces (Various Artists album featuring only music composed my Mozart

https://baseline.apple.com/training/evaluations/4/guidelines 3/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
14. [mozart] – Wolfgang Amadeus Mozart Essentials (Playlists featuring only the artist of the intended song)
15. [mozart] – Mozart Station (Composer station by Apple Music Classical)
16. [mozart] – Mozart: Requiem, K. 626 (Work aggregated in an album container by the intended composer)
17. [mozart] – Piano Concerto No. 21 in C Major, K. 467 (Work in a song container by the intended composer)
18. [mozart] – Le Concerto pour piano n°21 de Mozart par Daniel BARENBOIM (Work in a playlist container by the intended composer)
19. [mozart piano] – works by the composer in the intended genre/instrument (since the user didn't specify the exact work, this is considered an artist
navigational query rather than song)

Good: a piece of content which only partially satisfies the intent of the artist query and is related to the artist.
Songs by the intended artist coming from compilations of Various Artists albums
Artist Uploaded Content (AUC) of high quality
High-quality playlists that feature at least one song of the artist AND songs by other artists in similar genre. (For a visual representation of the rule,
see Playlist Rating table below.)
High-quality Various Artists albums that feature at least one song of the artist AND songs by other artists in similar genre.
"Inspired by" playlists for the intended artist by Apple Music or playlists created by the queried artist.
Radio shows/episodes where the queried artist performs or is interviewed.
Artist page for the other stage name of the queried artist
Band of the queried artist (note that members of the band when the query is for the band are rated Acceptable)

1. [lady gaga] – A playlist in the genre of pop: ‘10s Hits Essentials

2. [ed sheeran] – song Perfect by Ed Sheeran from the compilation album Internet Hits 2022
3. [notorious big] – a playlist in the genre of Hip-Hop/Rap: Can't Stop Won't Stop
4. [taylor swift] — Beats 1 Interview with Zane Lowe (AUC)
5. [taylor swift] — Playlist Inspired by Taylor Swift (this is an Apple Music Pop playlist
6. [taylor swift] — Playlist If You Like… Taylor Swift
7. [jlo] - Jennifer Lopez & Lin-Maniel Miranda: The Making of "Love Make the World Go Round" (AUC)
8. [nicky minaj] – a radio episode featuring Nicky Minaj
9. [byeonce] – THE CARTERS' artist page, and/or a song by THE CARTERS. This satisfies a secondary intent of the query and should be rated
good. Primary intent of the user is solo music by Beyoncé.
10. [billie eilish] – radio show me & dad radio by Billie Eilish
11. [2pac] – Makaveli artist page
12. [mozart] – The 50 Greatest Pieces of Classical Music (High-quality playlists/various artists albums that feature one work of the composer AND
works by other composers in similar genre)
13. [mozart]– Classical in Spatial Audio (High-quality playlists/various artists albums that feature one work of the composer AND works by other
composers in similar genre)

AUC example

Acceptable: a piece of content that only poorly satisfies the intent of the artist query.
Artist Uploaded Content of lower quality.
Unpopular pieces of content with the same name or title (the piece of content has no ratings, or the content has negative ratings), or matching part
of the title.
Content by secondary intent artist. (For example, for the query [pink] the content by unpopular artist Pink would be Acceptable instead of Excellent,
since the intent here is artist P!nk and only the content by that artist should be considered Excellent.)
Playlists/Various Artists albums that don’t feature any songs of the intended artist, BUT still fit in the genre of the intended artist (includes Apple
Music's Category pages). (For a visual representation of the rule, see Playlist Rating table below Song Navigational.)
Playlists/Various Artists albums that do features the artist BUT other songs genre is different to the intended artist’s genre. Note that if the playlist is
aggregated by year (e.g. charts), by activity (e.g. workout playlist), so the majority of the songs don't follow the genre of the intended artist, it is
considered different genre. (For a visual representation of the rule, see Playlist Rating table below Song Navigational.)
Low-quality playlists that feature at least one song of the artist AND songs by other artists in similar genre (note that high-quality is Good).
Pages of artists who have collaborated with the intended artist or are part of the band.
DJ remixes of the artist's songs, which are not recorded by the artists, but by DJs instead (Only when the query is for the artist and the returned
content is a DJ remix of the artist's song. When the query is for a DJ their songs should be considered "content by the artist" and rated Excellent)
Albums of other artists where the queried artist has a featured song.
Additionally, for classical music composers:
Covers and other versions of composer’s music (lullabies, modern instruments, etc)
Essentials Playlists for artists performing the music of composers
Music videos of the intended composer’s works

1. [BTS] -- a song titled "Daydream" by the artist J-Hope, one of the BTS boy band members
2. [BTS] -- The artist page for J-Hope, one of the BTS boy band members
3. [BTS] -- an album titled More Than You Know by B T S (Note that this is not the famous South Korean boy band, but an unpopular artist with a
matching name)
4. [notorious big] -- a playlist in the genre of Hip-Hop and R&B: Hip-Hop/R&B Hits: 1997 (poorly satisfies the intent as this is in the genre of R&B as
well as Hip-Hop)
5. [post malone] -- Top 100: USA (if the playlist features a song by Post Malone)
6. [taylor swift] — A song titled Taylor Swift by Dé Raj
7. [jlo] -- the artist page for Pitbull (Pitbull collaborated with Jennifer Lopez on On the Floor)
8. [The Weeknd] -- I Feel It Coming (feat. Daft Punk) [Mixed] The Weeknd a remix by Dj Voyst of the artist's popular song
9. [chirs brown] – album Detroit by Big Sean that includes a song featuring Chris Brown (the album is not a content of the queried artist per se, but it
still poorly satisfies the intent)
10. [mozart] – artist Mozart La Para (Unpopular pieces of content with matching name or title)
11. [mozart] – René Jacobs Essentials (playlist of a artist performing the music by composer)
12. [mozart] – Music From ‘Servant’: Season 2 (Playlists that do feature the composer BUT the genre of other tracks is different )
13. [mozart] – American Composers (Playlists that don’t feature any works of the intended composer, BUT still fit in the genre)
14. [chopin] – Chopin: Nocturne No. 20 in C-sharp Minor, Op.posth (Arr. for Violin and Piano by Milstein) (Music videos of the intended composer’s
works)

Unacceptable (Off-Topic): a piece of content that is not related to the song, and/or has no perceived relevance.
Piece of content that is not related to the artist, has no perceived relevance, or is likely to mislead users about its authorship.
Piece of content featuring similar names or titles, and media for which the connection with the artist is not readily apparent.
Playlists that don’t feature songs of the artist AND only partially fit in the genre of the intended artist

1. [lady gaga] -- Lady Saw

2. [lady gaga] -- A playlist in the genre of Jazz

https://baseline.apple.com/training/evaluations/4/guidelines 4/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
3. [the church] -- an album by Eric Church
4. [100 preguntas] - 100 Preguntas (feat. Ozun.A) by Yowell. The intended song is 100 Preguntas by Ozuna. Yowell is an unknown and unpopular
artist, and since they added “feat. Ozun.A” in the song title, this piece of content is likely to mislead users about its authorship, and will be rated as
Unacceptable: Off Topic.

Exception: Vocaloid artists

Vocaloid is a singing voice synthesizer software product. The voice of Vocaloid artists can be downloaded by anyone on the internet, so there is a lot
of unpopular content available featuring Vocaloid artists. Therefore, for Vocaloid artist queries, not all content featuring the intended artist should be
rated Excellent. Instead, only popular content should be rated Excellent. Less popular content should be rated Good, and unpopular content
should be rated Acceptable.

Song Navigational
Perfect: The particular song implied by the query, the result that satisfies primary intent.
1. [jamming bob marley] – the song "Jamming" by Bob Marley
2. [let it go] - the song Let It Go from Frozen by Idina Menzel
3. [ed sheeran perfect] – song Perfect from the artist's official album ÷ (Deluxe)
4. [yoü and i] – the song "Yoü And I" by Lady Gaga
5. [ronan taylor swift] – the single by Taylor Swift. If the query is a song and the returned result contains only that single, it is also eligible for the
rating Perfect.
6. [rockstar post malone clean] – Rockstar by Post Malone clean version. Please listen to the preview of the song if the query denotes clean or explicit
to make sure that the song matches the query intent.
7. *[respect] – "Respect" by Aretha Franklin
8. *[i will always love you] – "I Will Always Love You" by Whitney Houston
9. [Cardi B and Megan Thee Stallion] – the song WAP. For queries containing multiple artists: Do research to determine the most likely intent and rate
accordingly. For this example, in May 2022 the most likely intent is the song WAP, therefore this song is rated Perfect.
10. [home Phillip] – song by Phillip Phillips. Even though there are several popular pieces of content with the title “Home”, the intent of the query is clear
due to the artist name, so it should be rated Perfect.
11. [rachmaninoff piano concerto 2] –> Rachmaninov: Piano Concertos Nos. 2 & 3 (the user is looking for a full work by the composer, which consists of
multiple tracks aggregated in an album. Note that the album also includes other work, but is still rated Perfect)
12. [nutcracker] – album that contains all work movements of Nutcracker Tchaikovsky: The Nutcracker, Op. 71(the user is looking for a full work by the
composer, which consists of multiple tracks aggregated in an album)
13. [nutcracker] – Nutcracker Part 2: Dance of the Sugarplum Fairy (individual track which is part of the whole classical composition)
14. [nutcracker pas de deux] – Pas De Deux by London Symphony Orchestra (the user is looking for a specific movement in the composition)
15. [moonlight sonata] – Piano Sonata No.14, Op. 27 by Beethoven (note that classical music composition can have a numeric name or an AKA name,
like “moonlight sonata”, but both point to the same piece of work)
*Note: some song covers have become much more famous than the original, and can therefore be considered Perfect.

Also note that if a remix/live/acoustic/cover version is clearly more popular than the original, you can infer that it satisfies a primary intent. In that case,
the popular remix/live/acoustic/cover version can be rated Perfect, and the original version can be rated Excellent.
Note that for classical music queries, the composer query without a specific work title is considered Artist Navigational, not Song. Refer to
the example Unser Artist Navigational –> Excellent –> [mozart piano]

Excellent: A popular piece of content which satisfies a dominant intent of the song query.
Music video of the primary intent song.
Less popular versions of the primary intent song by the original artist: featuring artists/radio edits/etc. (Note that this is different from "Popular
remixes and covers of the intended song" that are rated Good.)
This includes Remastered versions of songs, if the audio-recording is slightly different (different track time, audio quality, etc). Please listen to the
original and Remastered tracks to compare. If the songs are equivalent in their audios, they can be rated Perfect.
Songs when the intent in not clear and there are multiple equally popular songs to satisfy the intent (there's no clear separation between primary vs
secondary intent).
The intended audio-equivalent song from a third-party compilation album.

1. [i will always love you] – Dolly Parton's original song from 1973. Made into a worldwide hit by Whitney Houston in 1992.
2. [respect] – the song originally written and released by Otis Redding. This song was made famous by Aretha Franklin a few years later.
3. [yoü and i] – the music video for "Yoü And I" by Lady Gaga
4. [ed sheeran perfect] – song Perfect by Ed Sheeran from the compilation album Internet Hits 2022
5. [home] – song by Phillip Phillips. There are several popular pieces of content with the title “Home” (Home by Phillip Phillips; Michael Bublé; Edward
Sharpe). Since we don’t know which one the user meant, the most popular & likely intents are eligible for Excellent.
6. [home] – song by Michael Bublé
7. [home] – song by Edward Sharpe & The Magnetic Zeros
8. [échame la culpa] – the music video of the intended song
9. [shallow] – song Shallow (Radio Edit) by Lady Gaga and Bradley Cooper
10. [nutcracker] – Video playlist The Nutcracker by Warner Classics

Good: A popular piece of content which only partially satisfies the intent of the song query: 1) content related to primary intent OR 2) popular
secondary intents.
1) Content related to primary intent song:
Popular remixes and covers of the intended song
Artist’s page of the intended song (Note: for classical music, composer is considered Good and artists performing the piece are considered
Acceptable)
Other music of the artist
Radio shows/episodes where the queried artist of the song performs or is interviewed
Artist Essentials playlists containing of the primary intent song's artist
High-quality AUC (Artist Uploaded Content) related to the song
High-quality playlists/Various Artists albums that feature the intended song AND songs by other artists in similar genre
Album of the intended song
Apple Music genre pages that fit primary intent of the query (genre/decade/mood/occasions)
2) Popular songs that satisfy clear secondary intent

1. [blank space] – the artist essentials playlist by Apple Music Taylor Swift Essentials
2. [yoü and i] – the artist page for Lady Gaga on Apple Music
3. [let it go] – the song Let It Go by Playboi Carti (this is a secondary intent song, primary being Let it Go by Idina Menzel)
4. [let it go] – the song Let It Go by James Bay (this is a secondary intent song, primary being Let it Go by Idina Menzel)
5. [let it go] – the artist page for Idina Menzel (artist’s page of the primary intent song)
6. [let it go] – the album Frozen, containing the primary intent Idina Menzel song (album of the primary intent song)
7. [let it go] – the playlist Disney Hits (features the intended song AND songs by other artists in similar genre)
8. [home] – The Piano Guys' rendition of the Phillip Phillips song
9. [home] – Phillip Phillips' album containing the song "Home"
10. [only the young] – A-List Pop by Apple Music (this playlist features the intended song and is in the genre of Pop, same as the intended song by
Taylor Swift; see Playlist Rating visual below)
11. [échame la culpa] – the artist page for Luis Fonsi
12. [échame la culpa] – Latin Pop Workout playlist (majority of the songs are in the same genre as the intended song)
13. [sleeping in the blacktop] – the artist page for Colter Wall
14. [nutcracker] – playlist Classical Essentials (High-quality playlists that feature the intended song AND songs by other artists in similar genre)
15. [nutcracker] – artist page of the composer Pyotr Ilyich Tchaikovsky
16. [moonlight sonata] – album Beethoven: Piano Sonatas (Moonlight, Pathétique & Appassionata)

Acceptable: A piece of content which only poorly satisfies the intent of the query: 1) unpopular secondary intents OR 2) poorly relevant results to
primary intent OR 3) content related to secondary intent song.
1) Unpopular secondary intents:

https://baseline.apple.com/training/evaluations/4/guidelines 5/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
Unpopular piece of content with the same title/name (the piece of content has no ratings, or the content has negative ratings), or matching
significant part of the title/name (beginning of the title or a well-known/commonly referred to part).
2) Poorly relevant content related to primary intent song:
Unpopular Remixes/Live/Karaoke/Acoustic versions of the primary intent song, and legitimate tributes.
Apple Music genre pages that poorly fit primary intent of the query (activity/mood/adjacent genres or decades)
Playlists/Various Artists albums that don’t feature the intended song BUT fit in the genre of the intended song. (For a visual representation of the
rule, see Playlist Rating table below.)
Playlists/Various Artists albums that do feature the song BUT other songs are different in genre. Note that if the playlist is aggregated by year (e.g.
charts), by activity (e.g. workout playlist), so the majority of the songs don't follow the genre of the intended song, it is considered different genre.
(For a visual representation of the rule, see Playlist Rating table below Song Navigational.)
Low-quality playlists/Various Artists albums that feature at least one song of the artist AND songs by other artists in similar genre (note that high-
quality is Good)
Playlists that feature the song from a third-party compilation (note: look at the cover and the album the song in the playlist is coming from)
For classical music: composer's artist page is considered Good and pages of artists performing the piece are considered Acceptable.
3) Content related to secondary intent song:
Any content related to the secondary intent song with rating Good (albums of that song, artist pages).

1. [let it go] – the artist page for Playboi Carti (artist of the secondary intent song with rating Good)
2. [let it go] – the album Playboi Carti (containing the secondary intent song with rating Good)
3. [let it go] – playlist Heartache (featuring the secondary intent song by James Bay AND songs by other artists in similar genre)
4. [let it go] – playlist Disney Princess (doesn't feature the primary intent song BUT fits in the genre of the intended song)
5. [let it go] – song Let It Go by Zac Brown Band (unpopular piece of content with the same title/name)
6. [you and i] – the music video "You and I Tonight" by Faber Drive
7. [rockstar post malone clean] – Rockstar by Post Malone explicit version. This version is not what the user requested, but it still slightly relevant.
8. [hypnotize biggie] – a playlist in the genres of Hip-Hop and R&B: Hip-Hop/R&B Hits: 1997. (Since this playlist is a mix of two different genres, it's
demoted to Acceptable. A playlist focused solely on Hip-Hop/Rap would be eligible for Good.)
9. [échame la culpa] – a less popular remix of the navigational song
10. [échame la culpa] – Behind the boards The Swaggernautz playlist (contains some Latin pop hits, but this piece of content is likely to only poorly
satisfy a secondary intent of the user)
11. [sleeping in the blacktop] – by Ritual Candles (an unpopular song matching the query. Primary intent is the song by Colter Wall.)
12. [home] – home andy yu (unpopular piece of content with matching to the query)
13. [nutcracker] – playlist Classical Christmas Hits (Playlists that do feature the song BUT other songs are different in genre, like charts)
14. [nutcracker] – Bonn Classical Philharmonic (artist page of the artist performing the classical piece)
15. [nutcracker] – Artist N2K NUTCRACKER 2000 (irrelevant unpopular matching title)
16. [moonlight sonata] – playlist Gustavo Dudamel Essentials (Playlists that don’t feature the intended song BUT fit in the genre of the intended song)
17. [fur elise] – Fur Elise (Remix) (Unpopular piece of content with the same title/name)

Unacceptable (Off-Topic): A piece of content that is not related to the song, and/or has no perceived relevance.
Content with no relevance to the query or matching insignificant part of the title. If you believe the user wouldn't be looking for that content with such
a query, rate it Unacceptable (Off-Topic).
Apple Music genre pages that are not relevant to the query
Any content related to the poorly relevant song with rating Acceptable (albums of that song, artist pages).

1. [jamming bob marley] – Just Jammin’ by Gramatik (the intent of the query is clear, so token matching is irrelevant in this context)
2. [sleeping in the blacktop] – The album Zen Garden 101 by Ritual Candles containing an Acceptable song. While the song itself is rated Acceptable,
the container of the song should be rated Unacceptable.
3. [sleeping in the blacktop] – the Artist page for Ritual Candles. This artist has an Acceptable song matching the query and is slightly relevant to the
query. (The primary intent of this query is the song by Colter Wall.) While the song itself is rated Acceptable, the artist page of the song should be
rated Unacceptable.
4. [let it go] – album You Get What You Give, featuring song Let It Go by Zac Brown Band (content related to the poorly relevant song with rating
Acceptable)

Song Navigational Intent-Ratings Guide (follow [let it go] examples under bullet points above)

Playlist Ratings for Song and Artist Navigational

https://baseline.apple.com/training/evaluations/4/guidelines 6/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine

Album Navigational
Perfect: The particular album implied by the query (EP, Album or "Single" album with more than one song)
1. [beatles box set] -- Beatles Box Set album
2. [life after death] -- Notorious B.I.G. album Life After Death
3. [444] -- the album 4:44 by JAY-Z

Excellent: A popular piece of content which satisfies a dominant intent of the album query (content of the intended album).
Songs and music videos from the intended album.
Artist Essentials playlists of the intended artist.
Radio episodes dedicated to the album.

1. [life after death] -- a song called Somebody's Gotta Die from the intended B.I.G. album
2. [dangerously in love] - music video of a song from the intended album
3. [1989] -- playlist by Apple Music Taylor Swift Essentials. This playlist contains 6 songs from the intended album.
4. [ed sheeran divide] -- a radio episode titled "Ed Sheeran: Songwriter", an Apple Music exclusive documentary about the making of Divide
5. [drake scorpion] -- OVO radio episode 65; Team OVO celebrates the release of Scorpion. (Note: OVO Sound is a record label founded by Drake.)

Good: A popular piece of content which only partially satisfies the intent of the query (content related to the album).
Album's artist page
High quality Artist Uploaded Content (AUC)
Playlists that fit in similar style/genre as the songs from the intended album (the playlist does not have to have songs from the album).
Other music by the artist.

1. [celebration album] - artist page for Madonna

2. [444] -- the artist page for JAY-Z
3. [life after death] — 90s Hip-Hop Essentials playlist by Apple Music

Acceptable: A piece of content which only poorly satisfies the intent of the query (unpopular secondary intents).
Also, unpopular pieces of content with the same title/name (the piece of content has no ratings, or the content has negative ratings)
Unpopular/lower quality Artist Uploaded Content (AUC).

1. [hands all over] - a song titled "Hands All Over You" by Kalin and Myles
2. [life after death] — an unpopular album by an artist named “Life After Death”
3. [444] -- Classic Animated Videos Playlist containing one song from the intended album

Unacceptable (Off-Topic): A piece of content that is not related to the album or has no perceived relevance. This includes pieces of content with
partially matching keywords in their titles, and content that is irrelevant to the album.
1. [beatles box set] -- "Ice Box". This result is unrelated to The Beatles.

Soundtrack Navigational
Note that a query like [every night in my dreams i see you titanic] should be classified as Lyrics, not Soundtrack, since it's referring to the intended
content by the lyrics (My Heart Will Go On by Celine Dion from the movie Titanic)

Perfect: The intended content implied by the query.

1. [the descendants soundtrack] – Descendants (Original TV Movie Soundtrack)
2. [the fast and the furious soundtrack collection] – The Fast and the Furious Soundtrack Collection
3. [deadpool soundtrack] – Deadpool (Original Soundtrack Album)
4. [friday the 13th main theme] — Friday the 13th Main Theme (feat. Jason Voorhees) [From Friday the 13th] - Single
5. [frozen 2] — Frozen 2 (Original Motion Picture Soundtrack) [Deluxe Edition]

Excellent: A piece of content which satisfies a dominant intent of the query.

Individual songs from the soundtrack compilation.
Versions/volumes other than the intended one by the query.

1. [back to the future soundtrack] – Back To the Future - Song from the intended soundtrack
2. [assasins creed] – Assassin's Creed Valhalla: The Ravens Saga (Original Soundtrack)
3. [descendants] – Descendants 3 (Original TV Movie Soundtrack) - The version is not specified in the search, so assume the user is looking for the
first version. Other versions/volumes would be Excellent results. Also use your market knowledge to determine which version the user is most likely
looking for.

Good: A popular piece of content which only partially satisfies the intent of the query.
Artist pages of the intended soundtrack.
Secondary intents (other versions, popular matching content)

1. [beauty and the beast] – Beauty and the Beast: The Broadway Musical (Original Broadway Cast Recording) - roadway musical would be secondary
intent
2. [bridgerton] – The Unofficial Bridgerton Musical

Acceptable: A piece of content which only poorly satisfies the intent of the soundtrack query.
This includes content of lower quality and unpopular pieces of content with the same name or title.

1. [frozen] – Frozen by Madonna & Sickick - a song with a matching title

2. [guardians of the galaxy] – Guardians Of The Galaxy, Vol.1 - Sleeping Lullabies for Babies Piano Covers

Unacceptable (Off-Topic): A piece of content that is not related to the query or has no perceived relevance.
1. [outer banks soundtrack] – Gabriel Yared (Artist page for Gabriel Yared, who has an unrelated song titled “The Outer Banks”)
2. [it soundtrack] – Purple Rain (no relevance to query)

Playlists
Note: Playlists need to have a minimum of 5 songs to be eligible for a Perfect rating. The highest possible rating for playlists with less than 5 songs is
Acceptable. In addition, all playlists must be rated for relevancy.

Playlist Navigational

Perfect: User is looking for a specific playlist (intent is very clear from the query)
1. [taylor swift deep cuts] – playlist by Apple Music "Taylor Swift: Deep Cuts"
2. [‘00s r&b essentials] – the intended playlist by Apple ’00s Hip-Hop/R&B Essentials
3. [kanye west essentials] – Kanye West Essentials by Apple Music Hip-Hop

Excellent: A piece of content which satisfies a dominant intent of the query (like video playlists satisfying the intent)
1. [kanye west essentials] – Kanye West Video Essentials by Apple Music Hip-Hop

Good: A popular piece of content which only partially satisfies the intent of the query.
Artist pages for the queried artist

https://baseline.apple.com/training/evaluations/4/guidelines 7/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
Songs by the intended artist of Essentials playlist
Songs from the intended playlist
Playlists that poorly satisfy the intent.

1. [get to know taylor swift] – artist page for Taylor Swift

2. [kanye west essentials] – songs by the intended artist are eligible for Good
3. [‘00s r&b essentials] — a ’90s R&B Essentials likely to only partially satisfy the intent
4. [rap life playlist] – California Breeze by Lil Baby (a song from the playlist)

Acceptable: A piece of content that is related to the query, but is unpopular or only poorly satisfies a secondary intent.

Unacceptable (Off-Topic): A piece of content that is not related to the query or has no perceived relevance.

Playlist Functional
Examples of Playlist Functional queries include [90s playlist], [party playlist], [workout playlist], etc. and express intent for playlists with a particular
function.

Perfect: Pieces of content which very clearly satisfy the intent of the query.
High quality playlists relevant to the query.
Activity/Mood pages relevant to the query.

1. [90s playlist] – a 90s Smash Hits playlists by Filtr

2. [dance party playlist] – Party Starters playlist by Apple Music Pop
3. [workout playlist] – Pure Workout by Apple Music
4. [workout playlist] – the Fitness Activity page containing relevant playlists (and radio stations).

Excellent: Piece of content which satisfies the intent of the query, but are a little more specific.
High-quality playlists in the genre with a specific activity/mood/era/occasion
Popular various artists' albums in the intended genre

1. [pop playlist] -- Apple Music playlist Pop Hits: 2020 (playlists in the intended genre that is a little specific on era/year)
2. [pop playlist] -- Apple Music playlist Workout Pop (playlists in the intended genre that is a little specific on activity)
3. [workout playlist] – Hip-Hop Workout by Apple Music (for a broad query, this Hip-Hop playlist is slightly specific, but still highly relevant)
4. [90s playlist] – 90s Hits album by Various Artists
5. [pop playlist] -- NOW That’s What I Call Music! Vol. 73 (a popular album by various artists)

Good: A piece of content which only partially satisfies the intent of the query, but is still high-quality
Radio stations in the intended genre
Music videos satisfying the intent
Individual popular artist’s compilations (albums and playlists) in that genre.

1. [90s playlist] -- The '90s Radio station by Apple Music

2. [rock playlist] -- Nirvana Essentials playlist (popular individual's artist playlist)

Acceptable: Pieces of content that poorly satisfies a secondary intent of the query.
Artist pages in the intended genre
Single songs / covers of a song / music videos in the intended genre
Artist uploaded content in the intended genre
Unpopular matching titles that can poorly satisfy the intent of the query
Individual unpopular artist’s compilations (albums and playlists) in that genre (consider storefront popularity as well)

1. [party playlist] -- Beyoncé's artist page

2. [90s playlist] -- the song "Nothing Compares 2 U" by Sinéad O'Connor

Unacceptable (Off-Topic): A piece of content that is not part of the specified genre.

Radio
** Radio Navigational/Functional classification types are being deprecated soon. Follow the current rating guideline until the new
classification types have been implemented. **

Radio Navigational
Queries that refer to a specific radio station. Example queries include [beats 1], [spa radio], [pure fuego], etc.

Examples:
Perfect: User is looking for a specific radio (intent is very clear from the query)
1. [charting now radio] -- Charting Now radio by Apple Music
2. [queen radio nicki] -- the "Queen Radio" show by Nicki Minaj
3. [bbc] -- BBC World Service UK
4. [kmle] -- KMLE Country 1079
5. [kool fm] -- KOOL-FM (this is the intended radio station based in Phoenix, Arizona)

Excellent: A piece of content which very clearly satisfies the intent of the query:
1. [queen radio nicki] -- an episode from the "Queen Radio" show by Nicki Minaj

Good: A piece of content which satisfies a secondary intent.

1. [bbc] -- the song by Jay-Z
2. [queen radio nicki] -- the artist page for Nicki Minaj

Acceptable: a piece of content that poorly satisfies a secondary intent of the query. For example, unpopular pieces of content with the same
title/name (the piece of content has no ratings, or the content has negative ratings), and less popular radio stations based outside of the market you’re
evaluating for.
1. [kool fm] -- Kool FM (Live at Funkhaus Berlin)
2. [kool fm] -- Kool FM (song) by Four Tet

Unacceptable (Off-Topic): a piece of content that is not related to the intended radio.
1. [kmle] -- KYLE the artist

Radio Functional
Queries that include a functional query for a genre/era/mood/style of music and the word ‘radio’ in the string. Example queries include [country music
radio], [k-pop songs radio], [relaxation music radio], etc.

Perfect: A radio station that satisfies a primary intent of the functional query, and is of high quality.
1. [country music radio] -- Pure Country Radio by Apple Music
2. [k-pop songs radio] -- K-pop Radio by Apple Music
3. [relaxation music radio] -- Ambi Nature Radio

Excellent: Radio stations playing music in the intended genre, but from a very specific era or sub-genre (only plays music in the genre from the 80’s,
90’s etc.).

https://baseline.apple.com/training/evaluations/4/guidelines 8/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
1. [country music radio] — ‘80’s & ’90’s Country Radio by Apple Music

Good: A piece of content which satisfies a secondary intent. For example, high quality playlists and Various Artists albums in the intended
genre/era/mood/style of music.
1. [k-pop songs radio] -- The A-List: K-Pop Playlist
2. [classic rock radio] -- Classic Rock various artist album

Acceptable: A piece of content that poorly satisfies a secondary intent of the query such as artist pages, single songs, or albums by individual artists
in the intended genre/era/mood/style of music.
1. [country music radio] — The artist page for Blake Shelton
2. [k-pop songs radio] -- The song Likey by Twice
3. [relaxation music radio] -- The song Moonlight by Relaxation Music

Unacceptable (Off-Topic): A piece of content that does not satisfy any of the above conditions.

** Use the following radio ratings once the new radio classification types are visible. **

Broadcast Radio
Queries that refer to conventional hosted live stations, often associated with a radio network. This includes local, satellite and internet radio stations.

Examples:
Perfect: User is looking for a specific live radio (intent is very clear from the query). For queries that only specify the frequency, several poplar radio
stations that are based in the same storefront and satisfy the intent can be rated Perfect.
1. [106.5] -- 106.5 The Beat (this is the intended radio station based in Richmond, Virginia)
2. [106.5] – 106.5 Kiss FM (popular radio station based in Huntsville, AL)
3. [106.1] – 106.1 KMEL (popular radio station based in San Fransisco)
4. [106.1] – HITS 106.1 (popular radio station based in Seattle/Tacoma)
5. [fox sports radio] -- Fox Sports Radio
6. [bbc radio world service] -- BBC World Service

Excellent: A piece of content which very clearly satisfies the intent of the query.
1. [bbc radio] – BBC Radio 4 (since the user is not specifying the number, assume the intent is first)

Good: A piece of content which satisfies a secondary intent.

1. [bbc] – the song by Jay-Z

Acceptable: A piece of content that poorly satisfies a secondary intent of the query. For example, unpopular pieces of content with the same
title/name (the piece of content has no ratings, or the content has negative ratings), and less popular radio stations based outside of the market you’re
evaluating for.
1. [kool fm] – Kool FM (Live at Funkhaus Berlin)
2. [Radio Dunedin] – OAR FM Dunedin
3. [iHeart pop radio] – Pop Playlists on Apple Music

Unacceptable (Off-Topic): a piece of content that is not related to the intended radio.
1. [kmle] – KYLE the artist
2. [GB radio] – Lullabies editorial radio station
3. [106.5] – 106.7 K-LOVE Radio WKVK

Apple Music Hosted Radio

Queries that refer to Apple Music hosted live radio shows. There are only three Apple hosted radio stations: Apple Music 1, Apple Music Hits, and
Apple Music Country. These three radio stations have many different shows on their programming.

Perfect: The intended radio station or radio show.

1. [Apple Music hits] — Apple Music Hits
2. [Beats 1 radio] — Apple Music 1, previously called Beats 1.
3. [after school radio] — After School Radio

Excellent: A piece of content which very clearly satisfies the intent of the query. For example, an episode from the intended radio show.
1. [queen radio nicki] — an episode from the "Queen Radio" show by Nicki Minaj
2. [after school radio] — Mark Finds a Gemstone: an episode from the intended radio show

Good: A piece of content which satisfies a secondary intent. For example the artist page of the radio show host, or the playlist featured on the
intended radio show page.
1. [queen radio nicki] — The artist page for Nicki Minaj
2. [80s radio with Huey Lewis] — The artist page for Huey Lewis & The News
3. [back porch country radio] — The Back Porch Country playlist, featured on the intended radio show page.

Acceptable: A piece of content that poorly satisfies a secondary intent of the query. For example, unpopular pieces of content with the same
title/name, and less popular radio stations based outside of the market you’re evaluating for.
1. [the 411] — The artist page for 411, an unpopular and irrelevant artist.

Unacceptable (Off-Topic): A piece of content that is not related to the intended radio.
1. [radio] — The artist page for The Radio. The Apple Music 1 radio station should be returned.

Editorial Radio
Queries that refer to non-hosted radio stations, focusing on a certain genre/mood/era. This includes utterances with a functional query + radio in the
string, for example [pop songs radio], [rock songs radio].

Perfect: A radio station that satisfies a primary intent of the functional query, and is of high quality.
1. [country music radio] -- Pure Country Radio by Apple Music
2. [k-pop songs radio] -- K-pop Radio by Apple Music
3. [relaxation music radio] -- Ambi Nature Radio

Excellent: Radio stations playing music in the intended genre, but from a very specific era or sub-genre (only plays music in the genre from the 80’s,
90’s etc.).
1. [country music radio] — ‘80’s & ’90’s Country Radio by Apple Music

Good: A piece of content which satisfies a secondary intent. For example, high quality playlists and Various Artists albums in the intended
genre/era/mood/style of music.
1. [k-pop songs radio] -- The A-List: K-Pop Playlist
2. [classic rock radio] -- Classic Rock various artist album

Acceptable: A piece of content that poorly satisfies a secondary intent of the query such as artist pages, single songs, or albums by individual artists
in the intended genre/era/mood/style of music.
1. [country music radio] — The artist page for Blake Shelton
2. [k-pop songs radio] -- The song Likey by Twice
3. [relaxation music radio] -- The song Moonlight by Relaxation Music

Unacceptable (Off-Topic): a piece of content that is not related to the intended radio. The output should satisfy the intent of the query and be related
to the genre of the requested radio, not simply match the tokens.

https://baseline.apple.com/training/evaluations/4/guidelines 9/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
1. [jazz radio] -- Jauz artist page
2. [country music radio] --The song Radio by Lana Del Rey

Lyrics
Some of our users look for a specific song by entering a memorable, catchy and/or recurring lyric from a song as their search query. The ratings for
this classification follow the ratings for Song Navigational classification.
If the queries contain lyrics + artist/movie name, they should still be considered lyrics.

Perfect: If the query intent is very clear, the song matching that lyrics query.
1. [you used to call me on my cell phone] -- this points to the popular song Hotline Bling by Drake
2. [you used to call me on my cell phone drake] -- since the query is still mainly referring to the content (song Hotline Bling by Drake) by lyrics, it
should be classified as Lyrics
3. [every night in my dreams i see you titanic] -- the query refers to the song My Heart Will Go On by Celine Dion from the movie Titanic; even though
it uses the lyrics with the name of the movie, it should be classified as Lyrics, not Soundtrack
4. [i’m never gonna dance again] -- points to the popular song Careless Whisper by George Michael

Excellent: More than one likely intents that are equally popular and relevant.
Songs when the intent in not clear and there are multiple equally popular songs to satisfy the intent (there's no clear separation between primary vs
secondary intent).
Music video of the primary intent song implied by the lyrics.

1. [who do you think you are] -- Jar of Hearts by Christina Perry

2. [who do you think you are] -- the song by Spice Girls

Good: A popular piece of content which only partially satisfies the intent of the song query (content related to primary intent or popular secondary
intents).
Popular remixes and covers of the intended song implied by the lyrics
Artist’s page of the intended song implied by the lyrics
Album of the intended song implied by the lyrics
Artist Essentials playlists containing the primary intent song implied by the lyrics
Other music of the artist
High-quality playlists that feature the intended song AND songs by other artists in similar genre (note that low-quality is Acceptable)
Popular songs that satisfy clear secondary intent

1. [i’m never gonna dance again] -- Careless Whisper by Seether (a popular cover)
2. [you used to call me on my cell phone] —the artist page for Drake

Acceptable: a piece of content that poorly satisfies a secondary intent of the query, such as unpopular/unknown songs containing the queried for
lyrics, or unpopular title/artist name keyword matches.
1. [i lean not on my own understanding] -- Nothing I Hold Onto by Tracy Nitschke. Unpopular song by an unknown artist (around 500 views on
Youtube) containing the queried for lyrics.

Unacceptable (Off-Topic): A piece of content that does not satisfy any of the above conditions.

Genre (& Genre/Category)

** Genre classification type is being renamed to Genre/Category. Follow the current rating guideline for both. **

Music genre queries have broad intent and include queries like [heavy metal], [pop], [hip hop], [classical], [alternative], [hits] etc.
Notes:
For genre queries looking for noises instead of songs (e.g. white noise, nature sounds, ASMR) a song that's over 15 minutes long can be rated as
Perfect.
If the query is for hits/charts without a specific playlist name (i.e. not Playlist Navigational), take into account the storefront of the query and playlist's
creator. If the result returned is charts for another country, Apple “Top 100” playlists should be rated Acceptable, playlists by other users should be
rated Off-Topic.

Perfect: Pieces of content which very clearly satisfy the intent of the query.
High quality playlists
High quality radio stations
Apple Music category pages

1. [pop] -- Apple Music Pop

2. [pop] -- The A-List: Pop playlist by Apple Music
3. [pop] -- Pure Pop Radio by Apple Music
4. [pop] -- Apple Music pop Category page
5. [white noise] -- White Noise 1 Hour Long
6. [hits] -- Apple Music's charts Top 100: Italy (if the query is in Italian storefront)
7. [90s music] -- Apple Music Category page for 90s music
8. [90s music] -- Apple Music playlist '90s Hits Essentials

Excellent: Piece of content which satisfies the intent of the query, but are a little more specific.
Playlists in the genre with a specific activity/mood/era/occasion
Popular various artists' albums in the intended genre (not including soundtracks)

1. [pop] – Apple Music playlist Pop Hits: 2020 (playlists in the intended genre that is a little specific on era/year)
2. [pop] – Apple Music playlist Workout Pop (playlists in the intended genre that is a little specific on activity)
3. [pop] – Apple Music playlist All I Want For Christmas is Pop (playlists in the intended genre that is a little specific on occasion)
4. [pop] – Apple Music playlist Heartbreak Pop (playlists in the intended genre that is a little specific on mood)
5. [hits] – Top 100: Global
6. [90s music] – Apple Music playlist '90s Alternative Essentials
7. [pop] – NOW That’s What I Call Music! Vol. 73 (a popular album by various artists)

Good: Popular pieces of content which only partially satisfies the intent of the genre.
Individual popular artist’s compilations (albums and playlists) in that genre.
Soundtrack compilations in form of Various Artists albums in that genre.
Category pages/playlists for adjacent decades.

1. [rock] – The Getaway by Red Hot Chili Peppers

2. [rock] – Nirvana Essentials playlist (popular individual's artist playlist)
3. [hits] – Måneskin Essentials playlist (if the query is in Italian storefront: popular individual's artist playlist)
4. [90s music] – Album Millennium by Backstreet Boys
5. [90s music] – Category page for 80s music (rated Good only because the decade is adjacent, 70s would be demoted to Acceptable)
6. [kids] – Soundtrack compilation Sing (Original Motion Picture Soundtrack Deluxe)

Acceptable: Pieces of content that poorly satisfies a secondary intent of the query.
Artist pages in the intended genre
Single songs / covers of a song / music videos in the intended genre
Artist uploaded content in the intended genre
Individual unpopular artist’s compilations (albums and playlists) in that genre (consider storefront popularity as well)
Unpopular matching titles that can poorly satisfy the intent of the query

1. [rock] -- the music video, "Let the Good Times Rock"

2. [rock] -- an album by "Weird Al" Yankovic

https://baseline.apple.com/training/evaluations/4/guidelines 10/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
3. [pop] -- Hey Jude by Assagai (an unpopular cover)
4. [pop] -- the song, “7 rings’ by Ariana Grande
5. [pop] -- Justin Bieber- sorry (LIVE)
6. [rock] -- The artist Rock
7. [hits] -- Apple Music's charts Top 100: Germany (if the query is in Italian storefront)
8. [90s music] -- song Ms. Jackson by Outkast

Unacceptable (Off-Topic): A piece of content that is not part of the genre.

1. [rock] -- a piano concerto by Beethoven
2. [rock] -- the artist Bob Marley
3. [hits] -- non-Apple Music's playlist Ballermann Hits 2022 (if the query is in Italian storefront)
4. [90s music] -- Apple Music playlist Hip-Hop Hits: 2020

Ambiguous
** Ambiguous classification type is being deprecated soon and being substituted with Ambiguous – Multiple Classifications & Ambiguous –
Intent Unclear. Follow the current rating guideline until the new classification types have been implemented. **

A query that is too ambiguous to identify a single primary intent. These queries may fit into several intent categories; for example the query [blue]
could refer to a mood, a song, or an artist called Blue. You should decide on the most likely classification for the query and use the rating for that
classification accordingly. Describe your reasoning in the comments.
Let's take query [blue] as an example. If it is considered a genre, a high quality playlist can be rated Perfect. If there is a very popular song with that
title, it can be rated Perfect as well. If there is a popular artist with the name Blue and you believe it would be a primary intent, the song by that artist
would be rated Excellent.

Perfect: A piece of content which is very likely to satisfy a primary intent of the ambiguous query. Only the most popular content will be eligible for this
rating
1. [disney] – Disney Hits (Playlist)
2. [blue] – Apple Category page Feeling Blue (if we consider the query to be a genre query, the category page is rated Perfect)
3. [blue] – Song Blue (Da Ba Dee) by Eiffel 65 (if we consider the query to be a song query, the most popular song that satisfies primary intent is
considered Perfect)

Excellent: A piece of content which clearly satisfies the intent of the query but is less popular or less known.
1. [disney] – Disney Piano Playlist

Good: A piece of content which only partially satisfies the intent of the query, but is still high-quality OR satisfies a secondary intent of the query OR
satisfies the intent of the query but is lower quality.
1. [disney] – secondary intent result by Jim Gaffigan
2. [blue] – artist Blue October (if we consider the query to be an artist query, this result is relevant through text match but not that popular, so would
only satisfy secondary intent)

Acceptable: A piece of content that is related to the query, but may be of poor quality, not popular, or poorly satisfies a secondary intent of the query.
1. [disney] – an unpopular song called “Disney” by Felly
2.

Unacceptable (Off-Topic): A piece of content that does not satisfy any of the above conditions.

Curator
Perfect -- the intended Curator page
1. [Spinnin’ Records] -- the curator page for Spinning Records
2. [boiler room] -- the curator page for Boiler Room
3. [tiktok] -- the curator page for TikTok

Excellent -- curated playlist, and albums by the intended Curator (regardless of popularity) are all considered Excellent results. Also, relevant and
high quality playlists from other reputable providers can receive an Excellent rating.
1. [Spinnin’ Records] -- Hardest Drops - by Spinnin' Records
2. [Spinnin’ Records] -- Compilation album by the curator Spinnin Records Best of Dance 2020
3. [boiler room] -- Boiler Room Radio: Extended Interviews - Pierre Kwenders
4. [boiler room] -- Boiler Room Playlist - Ep 3
5. [tiktok] -- TikTok Dance playlist by the intended curator
6. [tiktok] -- TikTok Songs playlist by Topsify. This is a highly relevant playlist from a well know curator, featuring the biggest TikTok viral hits.

Good — a piece of content which only partially satisfies the intent of the curator query such as Songs on a curated compilation album.
1. [Spinnin’ Records] -- Song on a compilation album by the Curator Day Or Night
2. [Spinnin’ Records] -- Song on a compilation album by the Curator Grapevine
3. [Spinnin’ Records] -- Song on a compilation album by the Curator Escape
4. [boiler room] -- Song on a compilation album by the Curator Y Is la Ves
5. [tiktok] -- Kesha's song Tik Tok. This song is from 2010 and no longer the primary intent of most users. Satisfies a secondary intent and is rated
Good.

Acceptable -- a piece of content which only poorly satisfies the intent of the curator query. This may include curator Uploaded Content of lower quality
and unpopular pieces of content with the same name or title (the piece of content has no ratings, or the content has negative ratings), or matching part
of the title.
1. [Spinnin’ Records] -- a song titled ‘Spinning Records‘
2. [Spinnin’ Records] -- an unknown song titled ‘Spinning Records‘
3. [tiktok] -- an unknown song by an artist titled Tik Tok

Unacceptable (Off-Topic) -- a piece of content that is not related to the Curator, has no perceived relevance, or is likely to mislead users about its
authorship. This also includes pieces of content featuring similar names or titles, and media for which the connection with the curator is not readily
apparent.
1. [Spinnin’ Records] -- Song of an album that is not by the curator Issues (feat. Ice Cube, Anderson .Paak & Dem Jointz)
2. [Spinnin’ Records] -- Song of an album that is not by the curator I'm the One (feat. Justin Bieber, Quavo, Chance the Rapper & Lil Wayne)
3. [boiler room] -- Song Boiler by Limp Bizkit
4. [tiktok] -- an artist named T.O.K

Record Label
Perfect -- The particular record label implied by the query.
1. [fat possum] — the record label page for Fat Possum Records
2. [b’in music] — the record label page for 相信音樂 (B’in Music in English)
3. [glassnote] — the record label page for Glassnote Records
4. [dead oceans] — the record label page for Dead Oceans

Excellent — Record label’s Curator’s page, and all content either curated, or produced by the intended record label. This includes Artist pages for
artists signed by the record label, Playlists, Albums, Songs, and Radio Shows produced, and/or curated by the record label.
1. [fat possum] — the curator page for Fat Possum Records
2. [dead oceans] — the curator page for Dead Oceans
3. [fat possum] — the playlist Fat Possum: Album of the Week by Fat Possum Records
4. [drag city] — the playlist Drag City For The Rest of Us by Drag City
5. [mau5trap] — the radio show mau5trap presents...

https://baseline.apple.com/training/evaluations/4/guidelines 11/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine
6. [604 records] — the album A 604 Records Christmas by Various Artists (album produced by 604 Records)
7. [604 records] — the song Mittens (Robert Orton Mix) by Carly Rae Jepsen, and other songs from the album 604 Records: Bundled Up produced by
604 Records

Good — a popular piece of content which only partially satisfies the intent of the query.
1. [drag city] — the song Drag City by Jan & Dean
2. [heavenly sweetness] — the EP album Heavenly Sweetness Loves Calypso Rose by Calypso Rose (this album isn’t produced by the intended
record label)

Acceptable -- a piece of content that is related to the query, but is unpopular or only poorly satisfies a secondary intent. This includes unpopular
pieces of content matching, or partially matching the query; and content related to the query.
1. [fat possum] — the song Fat Possum by Parry Music. This is not a popular song; and it’s not produced by the record label
2. [fat possum] — the artist page for Parry Music. The artist has a song called ‘Fat Possum’; and he’s not produced by the record label
3. [dead oceans] — the artist page for the unpopular artist Dead Oceans
4. [dead oceans] — the song Don’t Wait by unpopular artist Dead Oceans
5. [drag city] — the artist page for Jean & Dean
6. [drag city] — the song Drag City by Baga chipz Mbe (unpopular song)

Unacceptable (Off-Topic) -- a piece of content that is not related to the record label. , and content that is irrelevant to the record label.
1. [fat possum] — the album Bluegrass Trash by Fat Man Squeeze. This album is not relevant to the query, and is not produced by the record label
2. [fat possum] — the album Let Music & Bodies Unite by Rat Vs Possum. This album is not relevant to the query, and is not produced by the record
label
3. [dead oceans] — the playlist Dead Can Dance Essentials. The artist Dead Can Dance is not produced by the intended record label, and the band is
not relevant to the query

Video Navigational
A query that refers to a specific Apple Music tv show or episode, as well as music videos for a specific song. Some query examples are [carpool
karaoke], [planet of the apps], [sicko mode video], etc.

Perfect – Results that satisfy primary intent of the query.

The intended tv show container/episode
Specific song's music video implied by the query
Compilation of artist's videos implied by the query
1. [carpool karaoke] – Carpool Karaoke The Series
2. [sicko mode video] – The music video for the song Sicko Mode by Travis Scott
3. [carpool karaoke shakira] – The Carpool Karaoke episode featuring Shakira
4. [justin bieber music videos] – Justin Bieber Video Essentials
5. [peaches video] – Video Peaches by Justin Bieber

Excellent – A piece of content that satisfies a dominant intent of the query and is relevant to the query.
The intended music video’s song
Individual songs by the queried artist.
Single episodes from the intended Apple Music TV show when the intent is for the full show, and vice versa.
1. [sicko mode video] – The song Sicko Mode by Travis Scott
2. [toosie slide music video] – The song Toosie Slide by Drake
3. [carpool karaoke] – Carpool Karaoke Shakira & Trevor Noah episode
4. [carpool karaoke shakira] – Carpool Karaoke The Series
5. [justin bieber music videos] – Individual song's video Peaches by Justin Bieber
6. [peaches video] – Song Peaches by Justin Bieber (the intended music video’s song)

Good – A piece of content that satisfies a secondary intent.

Artist page for the intended music video’s artist
Other content by the artist: songs, albums, and music videos from the intended artist.
Video of songs that could satisfy secondary intent of the query
1. [sicko mode video] – The music video for the song Yosemite by Travis Scott - Another song/video by the intended artist
2. [sicko mode video] – The artist page for Travis Scott
3. [justin bieber music videos] – Artist page for Justin Bieber
4. [peaches video] – Album Justice for Justin Bieber (album containing the song)
5. [peaches video] – Video of song Peaches by KAI (video of songs that could satisfy secondary intent of the query)
6. [peaches video] – Video of song Anyone by Justin Bieber (other content by the artist)

Acceptable — A piece of content which only poorly satisfies the intent of the video query.
This includes content of lower quality and unpopular pieces of content with the same name or title.
Playlists with the intended song/content
1. [sicko mode video] — Sicko Mode [B-Mix] music video by Bagstheboss (this is an unpopular cover of the intended song)
2. [peaches video] – Playlist Pop Drive (playlist with the queried song and others in similar genre)

Unacceptable (Off-Topic) – a piece of content that is not related to the query, has no perceived relevance, or is likely to mislead users about its
authorship.

Music Search v3 Guidelines

Introduction

What is Apple Music?

The importance of your wo…

Query Research

Query Intent

Primary and Secondary Int…

Query Type

Misspelled Queries

Results Research

User's Intent

Popularity

Recency

Tips & tricks

Preview pages

Singles (Album Container)

Rating Non-English Queries

Rating New Releases

Seasonal Results

Classical Music Queries

Problem: Other

https://baseline.apple.com/training/evaluations/4/guidelines 12/13
10/28/23, 8:07 PM Guidelines for Search - Apple Music Training — BaseLine

Unacceptable: Other

Mandatory Comment

Relevance Ratings based on…

Artist Navigational

Song Navigational

Album Navigational

Soundtrack Navigational

Playlists

Playlist Navigational

Playlist Functional

Radio

Radio Navigational

Radio Functional

Broadcast Radio

Apple Music Hosted Radio

Editorial Radio

Lyrics

Genre (& Genre/Category)

Ambiguous

Curator

Record Label

Video Navigational

Copyright © 2023 Apple Inc. All rights reserved. Apple Confidential.

https://baseline.apple.com/training/evaluations/4/guidelines 13/13