19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

Search - Music v4 Training

Guidelines for Search - Music v4 Training

Updated:
August 2024: Task comment example added.
July 2024: Clarification on bpm (beats per minute), Artist stations for Artist Navigational, and clean vs. explicit results.
March 2024: Clarification on "similar artists" - similar artists do NOT include the main artist (Artist Functional)
March 2024: Clarification that Artist Essentials playlists should be rated Good for "Genre/Category" queries if all songs fall into the requested queries
March 2024: Clarification on partial playlist matches
April 2024: New Classification Type added to Guidelines (Song Functional)

Introduction
In this document, we explain relevance rating guidelines for Apple Music Search. You will use the BaseLine tool to make these ratings.

What is Apple Music?

Apple Music is a music and video streaming service developed by Apple. Users select music to stream to their device on-demand, or they can listen to existing, curated playlists. The service also includes the Internet
radio station Apple Music 1 (previously known as Beats 1), which broadcasts live to over 100 countries 24 hours a day. Some highlights of what Apple Music offers to the users :
Stream 100 million songs ad‑free.
Download your favorite tracks. Play them offline.
Get exclusive and original content.
Listen across all of your devices.
Sing along, tap ahead, or just listen with lyrics view.
Listen live to local radio stations from around the world.
Discover songs you’ll love from music picked just for you.
Tap into new music with curated playlists from our editors.

The importance of your work as a Rater

The data we receive from you in the form of high quality relevance judgments will be used to build and improve artificial intelligence systems such as search algorithms and machine learned rankers that power the user
experience for Apple Music users.
Our ultimate goal is to surprise and delight our customers by improving search quality and enhancing customer satisfaction, and you play an important role in this.

Ask yourself “Why is this particular result returned for this query? Is this relevant?” Stay curious and do thorough research to understand the relationship between a query and the result.
Your attention to detail, research and language skills as well as your cultural knowledge of the market are all critical to the success of our projects. Please keep in mind that your tasks will be spot-checked for quality, and
measured against those of your peers. If your accuracy rate is consistently high enough, you may be qualified to work as an auditor who gives feedback to your peers.

Query Research
Query Intent
We expect you to use search resources such as Bing, Google, Yahoo, YouTube, Genius as well as Apple Music to help you understand the intent of each query. Searching "[query] apple music" or "[query] iTunes" will
help narrow down the intent. More on User's Intent in the Results Research section below.

Primary and Secondary Intent

The primary intent of a query is the most likely intent, i.e., the intent of most users who enter that query in Apple Music.
A secondary intent, on the other hand, is less likely, or would be a less popular intent compared to a primary one. A secondary intent could be:
content relevant to a smaller group of users than for the primary intent
lower quality/lesser known content such as unpopular covers, remixes from unknown artists.

Query Type
Each query is assigned a ‘Query type’. On the left hand side in BaseLine, inside the input metadata section you will find the classification of the query. The classification is already set (you do not have to classify the query
yourself).
You should rate the content relevancy based on (1) the classification of the query and (2) how relevant is the content in relation to the query and the query type.

The ‘Query type’ aims to help you determine the primary intent of the query. Please also consider potential secondary intents when doing your ratings.
Should you disagree with the ‘Query type’ classification, please explain in the comment section why it should be different. You can then apply the appropriate rating scale to complete your judgement.
Example Comment: “I believe that [red] shall be classified as Album rather than Soundtrack. I used this query type for the rating.”

Misspelled Queries
Some queries are misspelled by our users. When this is the case, assume the query intent is for the correctly spelled query.
If the query is [the weekend] or [lasy gaga], the user likely means [the weeknd] and [lady gaga]. Use your best judgement.
Missing words like feat., x, ft. in Artist Queries (like [nicki minaj lil baby] instead of [nicki minaj ft. lil baby]) would not be considered spelling errors, as there is nothing that's misspelled.

Results Research
The results are generally rated based on the query type (section: Relevance Ratings based on Query Type). However, there are three general aspects to consider when rating any result: user's intent, popularity and
recency.

User's Intent
You will come across more ambiguous queries with multiple intents (like [adore you]) or queries with a very narrow specified intent (like [adore you jessie]). For queries with unclear primary intent, popularity and recency
of returned content plays a bigger role. For queries with narrow clear intent, only the relevance of the result plays a role in ratings. See examples of two search queries and their ratings below.

Example 1: [adore you] –> The user’s intent is not clear/specific, since there are multiple songs with the title [adore you]. Therefore, we base the ratings of the results on popularity to satisfy the intent for the majority of
users.
The song ‘Adore You’ by Harry Styles > Perfect
The song ‘Adore You’ by Miley Cyrus > Excellent
Harry Styles’ Artist Page > Good
The song ‘Adore You’ by Jessie Ware > Acceptable
The album ‘What’s your pleasure?’ by Jessie Ware (containing the song ‘Adore You’) > Off Topic
Jessie Ware’s Artist Page > Off Topic
Example 2: [adore you jessie] –> This query specifies the artist along with the song name, so we are now sure of the intent. Therefore, the ratings are based only on relevancy to the query, disregarding popularity.
The song ‘Adore You’ by Jessie Ware > Perfect
The album ‘What’s your pleasure?’ by Jessie Ware (containing the song ‘Adore You’) > Good
Jessie Ware’s Artist Page > Good
Other music by Jessie Ware > Good
HAAi’s Remix of ‘Adore You’ by Jessie Ware > Acceptable
The song ‘Adore You’ by Harry Styles > Off Topic
The song ‘Adore You’ by Miley Cyrus > Off Topic
Harry Styles’ Artist Page > Off Topic

Popularity
If the user's intent is clear and narrow (like [adore you jessie] in the section above), popularity and recency doesn't play a role in ratings.
For queries with unclear intents, popularity of results helps us understand how many users we would satisfy with the following result. In other words, more popular content would be relevant to a larger group of users, so
the result is more likely to satisfy ambiguous intent.
There are multiple resources you can check to understand popularity of content in your storefront: Bing, Google, Yahoo, YouTube, Genius, Apple Music. Please make sure you use at least two resources before
making your decision.

Google
You can use Google to research the popularity of returned content. Searching "[query] apple music" or "[query] iTunes" will help narrow down the intent.

YouTube
1. Total Views on YouTube
To assess popularity of the result, you can look at the number of views for the song/artist (make sure to sum the number of views for the song or artist if there are multiple relevant videos for that content).

2. Average Monthly Views Since Upload (AMVSU)

This metric helps us look at popularity of content, taking into account recency as well.
If there are 2 popular songs with matching title and relatively similar number of average views, both songs can be eligible for Excellent. The metric we adopt is Average Monthly Views Since Upload (AMVSU) = Total
Number of Views ÷ Months since release. This metric can also help you decide between primary intent (Perfect) and secondary intent (Good). See example below.
Example: [freedom] –> The query has a slightly ambiguous intent that can refer to Pharrell Williams' or Jon Batiste‘s songs.
Pharrell William's song has 127M views on YouTube over the last 7 years. AMVSU –> 127÷84 = 1.5M views per month since upload
Jon Batiste‘s song has 14M views over the last year. AMVSU –> 14÷12 = 1.2M views per month since upload.
Therefore, even though the Pharrell William's song has more views overall, Jon Batiste‘s song can also qualify for the rating Perfect, given its recency and monthly popularity.

3. Artist's Top Songs

https://baseline.apple.com/training/evaluations/1048/guidelines 1/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

When researching popularity of the artist, you can type the name in search to check YouTube's artist page with top songs and albums listed below the name. Pay attention to average number of listens/views for a song to
understand artist's overall popularity.

Spotify
Spotify can be a good source for assessing popularity of content. You can use metrics such as:
1. Number of monthly listeners
2. Number of artist's followers
3. Plays on individual songs
4. Verified artists (with blue checkmark) is also a good indicator of artist's popularity.

Apple Music
You can use Apple Music's search results, top charts playlists and the page Top Songs by the artist as a supportingpopularity data point as well. Make sure to use other resources in your research as well.

Music/Search platforms relevant in your storefront

Popularity of content should be assess world-wide and for the specific storefront you're working with. If the content is not popular worldwide but is very popular in your storefront, the result will have higher rating due to
high popularity.

Recency
There are cases were a song was released in the last couple of months and might not have as many views as another older song with similar title. There is a possibility that the more recent song is the intent as it is
currently charting and/or trending. For cases like this, both the recent song and the older song with more views can both get a rating of Perfect/Excellent.
Example: [Let It Go]
Let It Go from the movie Frozen is the most popular song with matching title with 2.5 billion views. This Song would get a rating of Perfect, since it would be considered primary intent.
The song Let It Go by DJ Khaled ft. Justin Bieber and 21 Savage is less popular with 20 million views. However, the song is from a little over 1 month ago and was in the top charts. This is a possible primary intent and
therefore is eligible for Excellent along with Frozen song (as of 2021).
The song Let It Go by James Bay is another popular song (368 million views) with matching title. However, this song is from 6 years ago, and is less popular than the song from Frozen. We cannot use recency in this
case. The best rating for this song is Good because it satisfies secondary intent.

Tips & tricks

Preview pages
Each search result will have a link to an Apple Music Preview page. This preview page will provide you with more details about the result, and allow you to listen to snippets of songs. For Radio results, please judge
based on relevance. For example, if your query is [hip hop], and the returned result is Hip-Hop Radio by Apple Music, it's clear that the result is highly relevant, and there's no need for an Apple Music Subscription to
listen to the station to verify relevance. Should the query intent be for a country song, and the returned result is a Hip-Hop Radio, you can rate that result as Off Topic. Should a Country Radio station be returned for a
country song query, that radio can be rated as Good (again, without listening to the radio, just based on the genre of the song and the station).

Singles (Album Container)

Sometimes you will encounter song singles in album container. Here's an example of a Taylor Swift's song Ronan in an Album Single container. This can be considered equivalent to a song output and rated accordingly.
EP could be treated as albums if they contain multiple songs. If the result has "– Single" in the title but contains more than one song in the container (like an example of this album – Single with two songs in it) it should be
treated as album.

Rating Non-English Queries

Please note that Apple Music users query for content in multiple languages, not only in English.
All queries in all languages are valid search queries, and the results should be rated based on relevance and quality, just like with any other type of music query. Utilize translation tools and conduct thorough side
searches to understand the likely user intent.

Rating New Releases

When rating newly released content, you might see that the online popularity of the content (YouTube views, media, Google search results) is still low. Take into consideration the potential popularity of the item (based on
artist popularity, anticipation of the release, historical popularity, etc) and apply the rating accordingly. For example, a newly released content by a popular artist can be considered secondary intent and not be counted as
"unpopular matching content".

Seasonal Results
In some locales there are seasonal results which have additional relevance and popularity at specific times of year, for example Christmas songs are very popular in the US at the end of the year. Please consider the
seasonal popularity when rating results and consider increasing the rating due to seasonal popularity. E.g. Jazz Christmas playlist should be rated higher for a query [jazz] around Christmas time.

Classical Music Queries

Sometimes you can come across a query with a classical music intent (composer, composition name, playlist, etc).
For queries related to classical music, Composers are classified as Artists. All works composed by the queries composer can be treated as content by the artist. If not listed in the container itself, please research the
work title or album title to check whether the content is by the right composer.
Classical Work Name or Catalogue Number can be treated as Songs. Sometimes a classical composition can consistent of multiple tracks (movements). The composition can be aggregated in a Song, Album or
Playlist container. Work/Compositions: albums/playlists/song containers that consist the fullintended Work or only part(one movement) are considered Perfect. Please note that classical music composition can have a
numeric name (like “Piano Concerto No.14, Op.27) or AKA name (like “moonlight sonata”), but both point to the same piece of work. When coming across classical music query, please research the relevant composer
and composition online to asses relevance of the results.
Relevant sections (Song Navigational and Artist Navigational) provide additional examples and support for those.

Problem: Other
Use this rating when a problem or technical issue with the task in BaseLine that makes it impossible to judge relevance. For example a piece of content with an unavailable title, artist, and thumbnail image.
Problem: Other rating should not be used if the content is unavailable in Apple Music but it is still possible to determine relevance between input and output based on the result title or artist name and by completing a side
search on Google and/or on other streaming platforms such as Spotify, Deezer, etc.

Unacceptable: Other
Please disregard the rating Unacceptable: Other and only use Unacceptable: Off-Topic for Unacceptable results.

Mandatory Comment
Each rating must be explained in the comment box. Even if “optional” is indicated, rating comments are always mandatory.
The comment should be concise and must only explain why the rating chosen is the correct one in application of the guidelines.

Task comment example

Relevance Ratings based on Query Type

In relevance rating, we start with a query and score the relevance of content for that query. Let’s look at different query intents and some rating examples.

Artist Navigational

Perfect: a piece of content that clearly satisfies the intent of the query with clear intent.
The artist page for the band/musician/composer.
1. [lady gaga] – the artist page for Lady Gaga on Apple Music
2. [bruno mars] – the artist page for Bruno Mars
3. [bealtes] – the artist page for the band, The Beatles. (Note that the query is misspelled.)
4. [billie eilish] – the artist page for Billie Eilish
5. [the artist from the Super Bowl half time show 2023] – artist page for Rihanna

Excellent: content by the primary intent artist (in different music containers).
Songs, albums, and music videos uploaded by the artist, including featuring artists (regardless of popularity).
https://baseline.apple.com/training/evaluations/1048/guidelines 2/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

Important: The song needs to come from the artist's uploaded container (e.g. official album) or an official album of the feature song to be qualified for Excellent. Songs by the artist from compilation Various Artists
albums are rated Good. Soundtracks are not counted as Various Artists compilations and can be considered Excellent.
Note that covers/remixes of the artist's content are demoted to Acceptable, unless the query's intent is the DJ and the result is a remix by that DJ (in that case, it's considered content by the intended artist). For
more info, refer to examples 11 and 12.
Note that to qualify for Excellent the content needs to be by the primary intent artist, not simply match the query. For example, for the query [pink] the content by unpopular artist Pink would be Acceptable instead of
Excellent, since the intent here is artist P!nk and only the content by that artist should be considered Excellent.
Playlists featuring only the intended artist (Essentials, Deep Cuts, Next Steps).
Artist Stations.
Radio shows & episodes hosted by the queried for artist.
Songs that the artist is featured on.
Additionally for classical music queries:
Individual works by the composer (regardless of popularity or the artist performing). Works can include multiple tracks and can be in aggregated in songs/albums/playlists containers.
Composer station by Apple Music Classical (for composer Artist queries)
Playlists or Various Artists albums featuring only the music by composer (performed by other artists) .

1. [lady gaga] – Lady Gaga's album The Fame Monster

2. [sean paul] – song Baby Boy (feat. Sean Paul) by Beyoncé (song from the official album of another artist with the featured song)
3. [idina menzel] – song Let It Go from an official soundtrack album
4. [ed sheeran] – song Perfect from the artist's official album ÷ (Deluxe)
5. [bruno mars] – a music video featuring Bruno Mars
6. [taylor swift] – playlist by Apple Music Taylor Swift Essentials
7. [eminem] – the song Believe by Eminem
8. [nicky minaj] – Queen Radio hosted by Nicky Minaj
9. [billie eilish] – song Lovely by Billie Eilish and Khalid
10. [Doja Cat] – song I Like You (A Happier Song) [feat. Doja Cat] by Post Malone
11. [Ed Sheeran] – song For My Hand (feat. Ed Sheeran) [Mixed] (a remix by DJ Neptune of Ed Sheeran’s song) –> Acceptable
12. [Meduza] – song Bad Habits (MEDUZA Remix) [Mixed] (a remix by MEDUZA of Ed Sheeran’s song) –> Excellent
13. [mozart] – The 99 Most Essential Mozart Masterpieces (Various Artists album featuring only music composed my Mozart
14. [the artist from the Super Bowl half time show 2023] – Rihanna Essentials

Good: a piece of content which only partially satisfies the intent of the artist query and is related to the artist.
Songs by the intended artist coming from compilations of Various Artists albums
Artist Uploaded Content (AUC) of high quality
High-quality playlists that feature at least one song of the artist AND songs by other artists in similar genre. (For a visual representation of the rule, see Playlist Rating table below.)
High-quality Various Artists albums that feature at least one song of the artist AND songs by other artists in similar genre.
"Inspired by" playlists for the intended artist by Apple Music or playlists created by the queried artist.
Radio shows/episodes where the queried artist performs or is interviewed.
Artist page for the other stage name of the queried artist
Band of the queried artist (note that members of the band when the query is for the band are rated Acceptable)

1. [lady gaga] – A playlist in the genre of pop: ‘10s Hits Essentials

2. [ed sheeran] – song Perfect by Ed Sheeran from the compilation album Internet Hits 2022
3. [notorious big] – a playlist in the genre of Hip-Hop/Rap: Can't Stop Won't Stop
4. [taylor swift] — Beats 1 Interview with Zane Lowe (AUC)
5. [taylor swift] — Playlist Inspired by Taylor Swift (this is an Apple Music Pop playlist
6. [taylor swift] — Playlist If You Like… Taylor Swift
7. [jlo] - Jennifer Lopez & Lin-Maniel Miranda: The Making of "Love Make the World Go Round" (AUC)
8. [nicky minaj] – a radio episode featuring Nicky Minaj
9. [byeonce] – THE CARTERS' artist page, and/or a song by THE CARTERS. This satisfies a secondary intent of the query and should be rated good. Primary intent of the user is solo music by Beyoncé.
10. [billie eilish] – radio show me & dad radio by Billie Eilish
11. [2pac] – Makaveli artist page
12. [mozart] – The 50 Greatest Pieces of Classical Music (High-quality playlists/various artists albums that feature one work of the composer AND works by other composers in similar genre)
13. [mozart]– Classical in Spatial Audio (High-quality playlists/various artists albums that feature one work of the composer AND works by other composers in similar genre)

AUC example

Acceptable: a piece of content that only poorly satisfies the intent of the artist query.
Artist Uploaded Content of lower quality.
Unpopular pieces of content with the same name or title (the piece of content has no ratings, or the content has negative ratings), or matching part of the title.
Content by secondary intent artist. (For example, for the query [pink] the content by unpopular artist Pink would be Acceptable instead of Excellent, since the intent here is artist P!nk and only the content by that artist
should be considered Excellent.)
Playlists/Various Artists albums that don’t feature any songs of the intended artist, BUT still fit in the genre of the intended artist (includes Apple Music's Category pages). (For a visual representation of the rule, see
Playlist Rating table below Song Navigational.)
Playlists/Various Artists albums that do features the artist BUT other songs genre is different to the intended artist’s genre. Note that if the playlist is aggregated by year (e.g. charts), by activity (e.g. workout playlist), so
the majority of the songs don't follow the genre of the intended artist, it is considered different genre. (For a visual representation of the rule, see Playlist Rating table below Song Navigational.)
Low-quality playlists that feature at least one song of the artist AND songs by other artists in similar genre (note that high-quality is Good).
Pages of artists who have collaborated with the intended artist or are part of the band.
DJ remixes of the artist's songs, which are not recorded by the artists, but by DJs instead (Only when the query is for the artist and the returned content is a DJ remix of the artist's song. When the query is for a DJ their
songs should be considered "content by the artist" and rated Excellent)
Albums of other artists where the queried artist has a featured song.
Additionally, for classical music composers:
Covers and other versions of composer’s music (lullabies, modern instruments, etc)
Essentials Playlists for artists performing the music of composers
Music videos of the intended composer’s works

1. [BTS] — a song titled "Daydream" by the artist J-Hope, one of the BTS boy band members
2. [BTS] — The artist page for J-Hope, one of the BTS boy band members
3. [BTS] — an album titled More Than You Know by B T S (Note that this is not the famous South Korean boy band, but an unpopular artist with a matching name)
4. [notorious big] — a playlist in the genre of Hip-Hop and R&B: Hip-Hop/R&B Hits: 1997 (poorly satisfies the intent as this is in the genre of R&B as well as Hip-Hop)
5. [post malone] — Top 100: USA (if the playlist features a song by Post Malone)
6. [taylor swift] — A song titled Taylor Swift by Dé Raj
7. [jlo] — the artist page for Pitbull (Pitbull collaborated with Jennifer Lopez on On the Floor)
8. [The Weeknd] — I Feel It Coming (feat. Daft Punk) [Mixed] The Weeknd a remix by Dj Voyst of the artist's popular song
9. [chirs brown] – album Detroit by Big Sean that includes a song featuring Chris Brown (the album is not a content of the queried artist per se, but it still poorly satisfies the intent)
10. [mozart] – artist Mozart La Para (Unpopular pieces of content with matching name or title)
11. [mozart] – René Jacobs Essentials (playlist of a artist performing the music by composer)
12. [mozart] – Music From ‘Servant’: Season 2 (Playlists that do feature the composer BUT the genre of other tracks is different )
13. [mozart] – American Composers (Playlists that don’t feature any works of the intended composer, BUT still fit in the genre)
14. [chopin] – Chopin: Nocturne No. 20 in C-sharp Minor, Op.posth (Arr. for Violin and Piano by Milstein) (Music videos of the intended composer’s works)

Unacceptable (Off-Topic): a piece of content that is not related to the song, and/or has no perceived relevance.
Piece of content that is not related to the artist, has no perceived relevance, or is likely to mislead users about its authorship.
Piece of content featuring similar names or titles, and media for which the connection with the artist is not readily apparent.
Playlists that don’t feature songs of the artist AND only partially fit in the genre of the intended artist

1. [lady gaga] — Lady Saw

2. [lady gaga] — A playlist in the genre of Jazz
3. [the church] — an album by Eric Church
4. [100 preguntas] - 100 Preguntas (feat. Ozun.A) by Yowell. The intended song is 100 Preguntas by Ozuna. Yowell is an unknown and unpopular artist, and since they added “feat. Ozun.A” in the song title, this piece of
content is likely to mislead users about its authorship, and will be rated as Unacceptable: Off Topic.

Exception: Vocaloid artists

Vocaloid is a singing voice synthesizer software product. The voice of Vocaloid artists can be downloaded by anyone on the internet, so there is a lot of unpopular content available featuring Vocaloid artists. Therefore, for
Vocaloid artist queries, not all content featuring the intended artist should be rated Excellent. Instead, only popular content should be rated Excellent. Less popular content should be rated Good, and unpopular
content should be rated Acceptable.

Artist Functional
These queries look for artists that share a common theme. Examples are queries like “female artists”, “artists from the 90s” or “artists that won a grammy”. Please also note that queries that request similar artists to a
specific artist do NOT include the artists in the query. The user is looking for additional artists, and therefore (for example) Ed Sheeran content should be rated "Unacceptable: Off Topic" for a query like "similar artists to
Ed Sheeran".

Perfect: Very popular artists that match the exact intent of the query.
1. [European musicians] - Elton John
2. [pop artists] - Ed Sheeran
3. [female artists] - Taylor Swift
Excellent: Artists with medium popularity
1. [rock artists] - Pixies

https://baseline.apple.com/training/evaluations/1048/guidelines 3/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

2. [Canadian artists] - Anne Murray

3. [artists born before 2000] - Robbie Williams
Good: Unpopular Artists that match the exact intent of the query. Also content by popular artists that exactly match the intent of the query
1. [artists with multiple stage names] - Gold Digger by Kanye West
2. [artists that performed at Tomorrowland in 2022] - Axel Haube
3. [artists with more than 3 band members] - One by Metallica
Acceptable: Unrelated exact title matches as well as music by unpopular artists
1. [songwriters] - The right way by The Songwriters
2. [artists from Germany] - You by Noah Appelhans

Unacceptable (Off-Topic): Any other content.

1. [Canadian artists] - John Denver
2. [artists born before 1990] - Phoebe Bridgers
3. [artists similar to Ed Sheeran] - Ed Sheeran

Song Navigational

Perfect: The particular song implied by the query, the result that satisfies primary intent.
1. [jamming bob marley] – the song "Jamming" by Bob Marley
2. [let it go] - the song Let It Go from Frozen by Idina Menzel
3. [ed sheeran perfect] – song Perfect from the artist's official album ÷ (Deluxe)
4. [yoü and i] – the song "Yoü And I" by Lady Gaga
5. [ronan taylor swift] – the single by Taylor Swift. If the query is a song and the returned result contains only that single, it is also eligible for the rating Perfect.
6. [rockstar post malone clean] – Rockstar by Post Malone clean version. Please listen to the preview of the song if the query denotes clean or explicit to make sure that the song matches the query intent.
7. *[respect] – "Respect" by Aretha Franklin
8. *[i will always love you] – "I Will Always Love You" by Whitney Houston
9. [Cardi B and Megan Thee Stallion] – the song WAP. For queries containing multiple artists: Do research to determine the most likely intent and rate accordingly. For this example, in May 2022 the most likely intent is the
song WAP, therefore this song is rated Perfect.
10. [home Phillip] – song by Phillip Phillips. Even though there are several popular pieces of content with the title “Home”, the intent of the query is clear due to the artist name, so it should be rated Perfect.
11. [rachmaninoff piano concerto 2] –> Rachmaninov: Piano Concertos Nos. 2 & 3 (the user is looking for a full work by the composer, which consists of multiple tracks aggregated in an album. Note that the album also
includes other work, but is still rated Perfect)
12. [nutcracker] – album that contains all work movements of Nutcracker Tchaikovsky: The Nutcracker, Op. 71(the user is looking for a full work by the composer, which consists of multiple tracks aggregated in an album)
13. [nutcracker] – Nutcracker Part 2: Dance of the Sugarplum Fairy (individual track which is part of the whole classical composition)
14. [nutcracker pas de deux] – Pas De Deux by London Symphony Orchestra (the user is looking for a specific movement in the composition)
15. [moonlight sonata] – Piano Sonata No.14, Op. 27 by Beethoven (note that classical music composition can have a numeric name or an AKA name, like “moonlight sonata”, but both point to the same piece of work)
*Note: some song covers have become much more famous than the original, and can therefore be considered Perfect.

Also note that if a remix/live/acoustic/cover version is clearly more popular than the original, you can infer that it satisfies a primary intent. In that case, the popular remix/live/acoustic/cover version can be rated Perfect,
and the original version can be rated Excellent.
Note that for classical music queries, the composer query without a specific work title is considered Artist Navigational, not Song. Refer to the example Unser Artist Navigational –> Excellent –> [mozart piano]

My Tasks Training Excellent: A popular piece of content which satisfies a dominant intent of the song query. Yessenia Ortega
Music video of the primary intent song.
Less popular versions of the primary intent song by the original artist: featuring artists/radio edits/etc. (Note that this is different from "Popular remixes and covers of the intended song" that are rated Good.)
This includes Remastered versions of songs, if the audio-recording is slightly different (different track time, audio quality, etc). Please listen to the original and Remastered tracks to compare. If the songs are
equivalent in their audios, they can be rated Perfect.
Songs when the intent in not clear and there are multiple equally popular songs to satisfy the intent (there's no clear separation between primary vs secondary intent).
The intended audio-equivalent song from a third-party compilation album.

1. [i will always love you] – Dolly Parton's original song from 1973. Made into a worldwide hit by Whitney Houston in 1992.
2. [respect] – the song originally written and released by Otis Redding. This song was made famous by Aretha Franklin a few years later.
3. [yoü and i] – the music video for "Yoü And I" by Lady Gaga
4. [ed sheeran perfect] – song Perfect by Ed Sheeran from the compilation album Internet Hits 2022
5. [home] – song by Phillip Phillips. There are several popular pieces of content with the title “Home” (Home by Phillip Phillips; Michael Bublé; Edward Sharpe). Since we don’t know which one the user meant, the most
popular & likely intents are eligible for Excellent.
6. [home] – song by Michael Bublé
7. [home] – song by Edward Sharpe & The Magnetic Zeros
8. [échame la culpa] – the music video of the intended song
9. [shallow] – song Shallow (Radio Edit) by Lady Gaga and Bradley Cooper
10. [nutcracker] – Video playlist The Nutcracker by Warner Classics

Good: A popular piece of content which only partially satisfies the intent of the song query: 1) content related to primary intent OR 2) popular secondary intents.
1) Content related to primary intent song:
Popular remixes and covers of the intended song
Artist’s page of the intended song (Note: for classical music, composer is considered Good and artists performing the piece are considered Acceptable)
Other music of the artist
Radio shows/episodes where the queried artist of the song performs or is interviewed
Artist Essentials playlists containing of the primary intent song's artist
High-quality AUC (Artist Uploaded Content) related to the song
High-quality playlists/Various Artists albums that feature the intended song AND songs by other artists in similar genre
Album of the intended song
Apple Music genre pages that fit primary intent of the query (genre/decade/mood/occasions)
2) Popular songs that satisfy clear secondary intent

1. [blank space] – the artist essentials playlist by Apple Music Taylor Swift Essentials
2. [yoü and i] – the artist page for Lady Gaga on Apple Music
3. [let it go] – the song Let It Go by Playboi Carti (this is a secondary intent song, primary being Let it Go by Idina Menzel)
4. [let it go] – the song Let It Go by James Bay (this is a secondary intent song, primary being Let it Go by Idina Menzel)
5. [let it go] – the artist page for Idina Menzel (artist’s page of the primary intent song)
6. [let it go] – the album Frozen, containing the primary intent Idina Menzel song (album of the primary intent song)
7. [let it go] – the playlist Disney Hits (features the intended song AND songs by other artists in similar genre)
8. [home] – The Piano Guys' rendition of the Phillip Phillips song
9. [home] – Phillip Phillips' album containing the song "Home"
10. [only the young] – A-List Pop by Apple Music (this playlist features the intended song and is in the genre of Pop, same as the intended song by Taylor Swift; see Playlist Rating visual below)
11. [échame la culpa] – the artist page for Luis Fonsi
12. [échame la culpa] – Latin Pop Workout playlist (majority of the songs are in the same genre as the intended song)
13. [sleeping in the blacktop] – the artist page for Colter Wall
14. [nutcracker] – playlist Classical Essentials (High-quality playlists that feature the intended song AND songs by other artists in similar genre)
15. [nutcracker] – artist page of the composer Pyotr Ilyich Tchaikovsky
16. [moonlight sonata] – album Beethoven: Piano Sonatas (Moonlight, Pathétique & Appassionata)

Acceptable: A piece of content which only poorly satisfies the intent of the query: 1) unpopular secondary intents OR 2) poorly relevant results to primary intent OR 3) content related to secondary intent song.
1) Unpopular secondary intents:
Unpopular piece of content with the same title/name (the piece of content has no ratings, or the content has negative ratings), or matching significant part of the title/name (beginning of the title or a well-
known/commonly referred to part).
2) Poorly relevant content related to primary intent song:
Unpopular Remixes/Live/Karaoke/Acoustic versions of the primary intent song, and legitimate tributes.
Apple Music genre pages that poorly fit primary intent of the query (activity/mood/adjacent genres or decades)
Playlists/Various Artists albums that don’t feature the intended song BUT fit in the genre of the intended song. (For a visual representation of the rule, see Playlist Rating table below.)
Playlists/Various Artists albums that do feature the song BUT other songs are different in genre. Note that if the playlist is aggregated by year (e.g. charts), by activity (e.g. workout playlist), so the majority of the songs
don't follow the genre of the intended song, it is considered different genre. (For a visual representation of the rule, see Playlist Rating table below Song Navigational.)
Low-quality playlists/Various Artists albums that feature at least one song of the artist AND songs by other artists in similar genre (note that high-quality is Good)
Playlists that feature the song from a third-party compilation (note: look at the cover and the album the song in the playlist is coming from)
For classical music: composer's artist page is considered Good and pages of artists performing the piece are considered Acceptable.
3) Content related to secondary intent song:
Any content related to the secondary intent song with rating Good (albums of that song, artist pages).

1. [let it go] – the artist page for Playboi Carti (artist of the secondary intent song with rating Good)
2. [let it go] – the album Playboi Carti (containing the secondary intent song with rating Good)
3. [let it go] – playlist Heartache (featuring the secondary intent song by James Bay AND songs by other artists in similar genre)
4. [let it go] – playlist Disney Princess (doesn't feature the primary intent song BUT fits in the genre of the intended song)
5. [let it go] – song Let It Go by Zac Brown Band (unpopular piece of content with the same title/name)
6. [you and i] – the music video "You and I Tonight" by Faber Drive
7. [hypnotize biggie] – a playlist in the genres of Hip-Hop and R&B: Hip-Hop/R&B Hits: 1997. (Since this playlist is a mix of two different genres, it's demoted to Acceptable. A playlist focused solely on Hip-Hop/Rap would
be eligible for Good.)
8. [échame la culpa] – a less popular remix of the navigational song
9. [échame la culpa] – Behind the boards The Swaggernautz playlist (contains some Latin pop hits, but this piece of content is likely to only poorly satisfy a secondary intent of the user

https://baseline.apple.com/training/evaluations/1048/guidelines 4/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

10. [sleeping in the blacktop] – by Ritual Candles (an unpopular song matching the query. Primary intent is the song by Colter Wall.)
11. [home] – home andy yu (unpopular piece of content with matching to the query)
12. [nutcracker] – playlist Classical Christmas Hits (Playlists that do feature the song BUT other songs are different in genre, like charts)
13. [nutcracker] – Bonn Classical Philharmonic (artist page of the artist performing the classical piece)
14. [nutcracker] – Artist N2K NUTCRACKER 2000 (irrelevant unpopular matching title)
15. [moonlight sonata] – playlist Gustavo Dudamel Essentials (Playlists that don’t feature the intended song BUT fit in the genre of the intended song)
16. [fur elise] – Fur Elise (Remix) (Unpopular piece of content with the same title/name)

Unacceptable (Off-Topic): A piece of content that is not related to the song, and/or has no perceived relevance.
Content with no relevance to the query or matching insignificant part of the title. If you believe the user wouldn't be looking for that content with such a query, rate it Unacceptable (Off-Topic).
Apple Music genre pages that are not relevant to the query
Any content related to the poorly relevant song with rating Acceptable (albums of that song, artist pages).

1. [jamming bob marley] – Just Jammin’ by Gramatik (the intent of the query is clear, so token matching is irrelevant in this context)
2. [sleeping in the blacktop] – The album Zen Garden 101 by Ritual Candles containing an Acceptable song. While the song itself is rated Acceptable, the container of the song should be rated Unacceptable.
3. [sleeping in the blacktop] – the Artist page for Ritual Candles. This artist has an Acceptable song matching the query and is slightly relevant to the query. (The primary intent of this query is the song by Colter Wall.)
While the song itself is rated Acceptable, the artist page of the song should be rated Unacceptable.
4. [let it go] – album You Get What You Give, featuring song Let It Go by Zac Brown Band (content related to the poorly relevant song with rating Acceptable)
5. [rockstar post malone clean] – Rockstar by Post Malone explicit version. This version is not what the user requested, and it does not satisfy the factual requirements of the query. Explicit should be rated unacceptable if
the user is looking for a clean version and vice versa.

Song Functional
These queries look for songs that share a common theme. However, they are more specific than genres or broad categories, so it is unlikely that whole playlists or stations are available for these results. Examples are
queries like “songs by artists who were born in Kansas”, “songs that sing about the challenges of being unemployed”, “songs that won a grammy between 1999 and 2003”.

Perfect: Very popular music that matches the exact intent of the query.
Popular songs that exactly match the intent
Popular albums if all songs match the intent
1. [songs by artists who were born in New Mexico] - Take me home, Country Roads
2. [songs about owning a house] - Our House by Madness
3. [covers of Dolly Parton songs] - I will always love you by Whitney Houston
4. [songs with a 121 bpm] - Call Me Maybe by Carly Rae Jepsen *
Excellent: Music content with medium popularity
1. [songs about school] - Paul Simon - Me and Julio Down by the Schoolyard
2. [songs by Canadian artists] - Rock This Country by Shania Twain
3. [live songs from 2010s] - He is (Live) by Ghost
Good: Unpopular music content that matches the exact intent of the query and secondary intents.
Unpopular albums if all songs match the intent
Unpopular songs that exactly match the intent
Content related to the primary intent (e.g. artist pages, partially relevant playlists)
1. [songs that have a Spanish title] → Te amo by Apache 16
2. [songs by recorded in July 2021] - Sofa by LuaLua
Acceptable: Unrelated exact title matches
Unacceptable (Off-Topic): Any other content.
1. [songs by artists from Seattle] - Thunder by Imagine Dragons
2. [disney songs for a bbq] - Theme Song of Jurassic Park

* Note: For queries that are looking for bpm (beats per minute)/tempo, content that has a bpm of +-5 difference to the one specified in the query should be considered relevant.

Album Navigational

Perfect: The particular album implied by the query (EP, Album or "Single" album with more than one song)
1. [beatles box set] — Beatles Box Set album
2. [life after death] — Notorious B.I.G. album Life After Death
3. [444] — the album 4:44 by JAY-Z

Excellent: A popular piece of content which satisfies a dominant intent of the album query (content of the intended album).
Songs and music videos from the intended album.
Artist Essentials playlists of the intended artist.
Radio episodes dedicated to the album.

1. [life after death] — a song called Somebody's Gotta Die from the intended B.I.G. album
2. [dangerously in love] - music video of a song from the intended album
3. [1989] — playlist by Apple Music Taylor Swift Essentials. This playlist contains 6 songs from the intended album.
4. [ed sheeran divide] — a radio episode titled "Ed Sheeran: Songwriter", an Apple Music exclusive documentary about the making of Divide
5. [drake scorpion] — OVO radio episode 65; Team OVO celebrates the release of Scorpion. (Note: OVO Sound is a record label founded by Drake.)

Good: A popular piece of content which only partially satisfies the intent of the query (content related to the album).
Album's artist page
High quality Artist Uploaded Content (AUC)
Playlists that fit in similar style/genre as the songs from the intended album (the playlist does not have to have songs from the album).
Other music by the artist.

1. [celebration album] - artist page for Madonna

2. [444] — the artist page for JAY-Z
3. [life after death] — 90s Hip-Hop Essentials playlist by Apple Music

Acceptable: A piece of content which only poorly satisfies the intent of the query (unpopular secondary intents).
Also, unpopular pieces of content with the same title/name (the piece of content has no ratings, or the content has negative ratings)
Unpopular/lower quality Artist Uploaded Content (AUC).

1. [hands all over] - a song titled "Hands All Over You" by Kalin and Myles
2. [life after death] — an unpopular album by an artist named “Life After Death”
3. [444] — Classic Animated Videos Playlist containing one song from the intended album

Unacceptable (Off-Topic): A piece of content that is not related to the album or has no perceived relevance. This includes pieces of content with partially matching keywords in their titles, and content that is irrelevant to
the album.
1. [beatles box set] — "Ice Box“. This result is unrelated to The Beatles.

Album Functional
These queries look for albums that share a common theme. Examples are queries like “albums for partying”, “90s albums”, “rock albums”, “albums with a green cover”.

Perfect: Very popular albums that match the exact intent of the query.
1. [albums by European Musicians] - Future Nostalgia by Dua Lipa
2. [pop albums] - 1989 by Taylor Swift
3. [Albums with a number in the title] - 25 by Adele
Excellent: Albums with medium popularity
1. [rock albums] - Forever Changes by Love
2. [albums by Canadian artists] - Now by Shania Twain
3. [albums released in 2023] - Okemah Rising by the Dropkick Murphys
Good: Unpopular albums that match the exact intent of the query.
1. [albums that have a Spanish name] → El Sol by Average Rap Band
2. [albums by artists from Seattle] - Town Story by Dusty Cubby
3. [alternative albums] - All the King’s Ashes by Brother Townsend
Acceptable: Unrelated exact title matches
Unacceptable (Off-Topic): Any other content.
1. [albums from 2020] - The Fame by Lady Gaga
2. [albums by Scottish djs] - Listen by David Guetta

Soundtrack Navigational
Note that a query like [every night in my dreams i see you titanic] should be classified as Lyrics, not Soundtrack, since it's referring to the intended content by the lyrics (My Heart Will Go On by Celine Dion from the
movie Titanic)

Perfect: The intended content implied by the query.

https://baseline.apple.com/training/evaluations/1048/guidelines 5/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

1. [the descendants soundtrack] – Descendants (Original TV Movie Soundtrack)

2. [the fast and the furious soundtrack collection] – The Fast and the Furious Soundtrack Collection
3. [deadpool soundtrack] – Deadpool (Original Soundtrack Album)
4. [friday the 13th main theme] — Friday the 13th Main Theme (feat. Jason Voorhees) [From Friday the 13th] - Single
5. [frozen 2] — Frozen 2 (Original Motion Picture Soundtrack) [Deluxe Edition]

Excellent: A piece of content which satisfies a dominant intent of the query.

Individual songs from the soundtrack compilation.
Versions/volumes other than the intended one by the query.
1. [back to the future soundtrack] – Back To the Future - Song from the intended soundtrack
2. [assasins creed] – Assassin's Creed Valhalla: The Ravens Saga (Original Soundtrack)
3. [descendants] – Descendants 3 (Original TV Movie Soundtrack) - The version is not specified in the search, so assume the user is looking for the first version. Other versions/volumes would be Excellent results. Also
use your market knowledge to determine which version the user is most likely looking for.

Good: A popular piece of content which only partially satisfies the intent of the query.
Artist pages of the intended soundtrack.
Secondary intents (other versions, popular matching content)

1. [beauty and the beast] – Beauty and the Beast: The Broadway Musical (Original Broadway Cast Recording) - roadway musical would be secondary intent
2. [bridgerton] – The Unofficial Bridgerton Musical

Acceptable: A piece of content which only poorly satisfies the intent of the soundtrack query.
This includes content of lower quality and unpopular pieces of content with the same name or title.

1. [frozen] – Frozen by Madonna & Sickick - a song with a matching title

2. [guardians of the galaxy] – Guardians Of The Galaxy, Vol.1 - Sleeping Lullabies for Babies Piano Covers

Unacceptable (Off-Topic): A piece of content that is not related to the query or has no perceived relevance.
1. [outer banks soundtrack] – Gabriel Yared (Artist page for Gabriel Yared, who has an unrelated song titled “The Outer Banks”)
2. [it soundtrack] – Purple Rain (no relevance to query)

Playlists
Playlist Navigational

Perfect: User is looking for a specific playlist (intent is very clear from the query)
1. [taylor swift deep cuts] – playlist by Apple Music "Taylor Swift: Deep Cuts"
2. [‘00s r&b essentials] – the intended playlist by Apple ’00s Hip-Hop/R&B Essentials
3. [kanye west essentials] – Kanye West Essentials by Apple Music Hip-Hop

Excellent: A piece of content which satisfies a dominant intent of the query (like video playlists satisfying the intent)
1. [kanye west essentials] – Kanye West Video Essentials by Apple Music Hip-Hop

Good: A popular piece of content which only partially satisfies the intent of the query.
Artist pages for the queried artist
Songs by the intended artist of Essentials playlist
Songs from the intended playlist
Playlists that poorly satisfy the intent.

1. [get to know taylor swift] – artist page for Taylor Swift

2. [kanye west essentials] – songs by the intended artist are eligible for Good
3. [‘00s r&b essentials] — a ’90s R&B Essentials likely to only partially satisfy the intent
4. [rap life playlist] – California Breeze by Lil Baby (a song from the playlist)

Acceptable: A piece of content that is related to the query, but is unpopular or only poorly satisfies a secondary intent.

Unacceptable (Off-Topic): A piece of content that is not related to the query or has no perceived relevance.
hosted radio stations: Apple Music 1, Apple Music Hits, and Apple Music Country. These three radio stations have many different shows on their programming.

Playlist Functional
Examples of Playlist Functional queries include [90s playlist], [party playlist], [workout playlist], etc. and express intent for playlists with a particular function.

Perfect: Pieces of content which very clearly satisfy the intent of the query.
High quality playlists relevant to the query.
Activity/Mood pages relevant to the query.

1. [90s playlist] – a 90s Smash Hits playlists by Filtr

2. [dance party playlist] – Party Starters playlist by Apple Music Pop
3. [workout playlist] – Pure Workout by Apple Music
4. [workout playlist] – the Fitness Activity page containing relevant playlists (and radio stations).

Excellent: Piece of content which satisfies the intent of the query, but are a little more specific.
High-quality playlists in the genre with a specific activity/mood/era/occasion
Popular various artists' albums in the intended genre

1. [pop playlist] — Apple Music playlist Pop Hits: 2020 (playlists in the intended genre that is a little specific on era/year)
2. [pop playlist] — Apple Music playlist Workout Pop (playlists in the intended genre that is a little specific on activity)
3. [workout playlist] – Hip-Hop Workout by Apple Music (for a broad query, this Hip-Hop playlist is slightly specific, but still highly relevant)
4. [90s playlist] – 90s Hits album by Various Artists
5. [pop playlist] — NOW That’s What I Call Music! Vol. 73 (a popular album by various artists)

Good: A piece of content which only partially satisfies the intent of the query, but is still high-quality
Radio stations in the intended genre
Music videos satisfying the intent
Individual popular artist’s compilations (albums and playlists) in that genre.
Popular songs that fit the intended genre or category

1. [90s playlist] — The '90s Radio station by Apple Music

2. [rock playlist] — Nirvana Essentials playlist (popular individual's artist playlist)
3. [90s playlist] — the song "Nothing Compares 2 U" by Sinéad O'Connor

Acceptable: Pieces of content that poorly satisfies a secondary intent of the query.
Artist pages in the intended genre
Unpopular single songs / covers of a song / music videos in the intended genre
Artist uploaded content in the intended genre
Unpopular matching titles that can poorly satisfy the intent of the query
Individual unpopular artist’s compilations (albums and playlists) in that genre (consider storefront popularity as well)

1. [party playlist] — Beyoncé's artist page

Unacceptable (Off-Topic): A piece of content that is not part of the specified genre.

Radio
Broadcast Radio
Queries that refer to conventional hosted live stations, often associated with a radio network. This includes local, satellite and internet radio stations.

Examples:
Perfect: User is looking for a specific live radio (intent is very clear from the query). For queries that only specify the frequency, several poplar radio stations that are based in the same storefront and satisfy the intent can
be rated Perfect.
1. [106.5] — 106.5 The Beat (this is the intended radio station based in Richmond, Virginia)
2. [106.5] – 106.5 Kiss FM (popular radio station based in Huntsville, AL)
3. [106.1] – 106.1 KMEL (popular radio station based in San Fransisco)
4. [106.1] – HITS 106.1 (popular radio station based in Seattle/Tacoma)
https://baseline.apple.com/training/evaluations/1048/guidelines 6/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

5. [fox sports radio] — Fox Sports Radio

6. [bbc radio world service] — BBC World Service

Excellent: A piece of content which very clearly satisfies the intent of the query.
1. [bbc radio] – BBC Radio 4 (since the user is not specifying the number, assume the intent is first)

Good: A piece of content which satisfies a secondary intent.

1. [bbc] – the song by Jay-Z

Acceptable: A piece of content that poorly satisfies a secondary intent of the query. For example, unpopular pieces of content with the same title/name (the piece of content has no ratings, or the content has negative
ratings), and less popular radio stations based outside of the market you’re evaluating for.
1. [kool fm] – Kool FM (Live at Funkhaus Berlin)
2. [Radio Dunedin] – OAR FM Dunedin
3. [iHeart pop radio] – Pop Playlists on Apple Music

Unacceptable (Off-Topic): a piece of content that is not related to the intended radio.
1. [kmle] – KYLE the artist
2. [GB radio] – Lullabies editorial radio station
3. [106.5] – 106.7 K-LOVE Radio WKVK

Apple Music Hosted Radio

Queries that refer to Apple Music hosted live radio shows. There are only three Apple hosted radio stations: Apple Music 1, Apple Music Hits, and Apple Music Country. These three radio stations have many different
shows on their programming.

Perfect: The intended radio station or radio show.

1. [Apple Music hits] — Apple Music Hits
2. [Beats 1 radio] — Apple Music 1, previously called Beats 1.
3. [after school radio] — After School Radio

Excellent: A piece of content which very clearly satisfies the intent of the query. For example, an episode from the intended radio show.
1. [queen radio nicki] — an episode from the "Queen Radio" show by Nicki Minaj
2. [after school radio] — Mark Finds a Gemstone: an episode from the intended radio show

Good: A piece of content which satisfies a secondary intent. For example the artist page of the radio show host, or the playlist featured on the intended radio show page.
1. [queen radio nicki] — The artist page for Nicki Minaj
2. [80s radio with Huey Lewis] — The artist page for Huey Lewis & The News
3. [back porch country radio] — The Back Porch Country playlist, featured on the intended radio show page.

Acceptable: A piece of content that poorly satisfies a secondary intent of the query. For example, unpopular pieces of content with the same title/name, and less popular radio stations based outside of the market you’re
evaluating for.
1. [the 411] — The artist page for 411, an unpopular and irrelevant artist.

Unacceptable (Off-Topic): A piece of content that is not related to the intended radio.
1. [radio] — The artist page for The Radio. The Apple Music 1 radio station should be returned.

Editorial Radio
Queries that refer to non-hosted radio stations, focusing on a certain genre/mood/era. This includes utterances with a functional query + radio in the string, for example [pop songs radio], [rock songs radio].

Perfect: A radio station that satisfies a primary intent of the functional query, and is of high quality.
1. [country music radio] — Pure Country Radio by Apple Music
2. [k-pop songs radio] — K-pop Radio by Apple Music
3. [relaxation music radio] — Ambi Nature Radio

Excellent: Radio stations playing music in the intended genre, but from a very specific era or sub-genre (only plays music in the genre from the 80’s, 90’s etc.).
1. [country music radio] — ‘80’s & ’90’s Country Radio by Apple Music

Good: A piece of content which satisfies a secondary intent. For example, high quality playlists and Various Artists albums in the intended genre/era/mood/style of music.
1. [k-pop songs radio] -- The A-List: K-Pop Playlist
2. [classic rock radio] — Classic Rock various artist album

Acceptable: A piece of content that poorly satisfies a secondary intent of the query such as artist pages, single songs, or albums by individual artists in the intended genre/era/mood/style of music.
1. [country music radio] — The artist page for Blake Shelton
2. [k-pop songs radio] -- The song Likey by Twice
3. [relaxation music radio] — The song Moonlight by Relaxation Music

Unacceptable (Off-Topic): a piece of content that is not related to the intended radio. The output should satisfy the intent of the query and be related to the genre of the requested radio, not simply match the tokens.
1. [jazz radio] — Jauz artist page
2. [country music radio] —The song Radio by Lana Del Rey

Lyrics
Some of our users look for a specific song by entering a memorable, catchy and/or recurring lyric from a song as their search query. The ratings for this classification follow the ratings for Song Navigational classification.
If the queries contain lyrics + artist/movie name, they should still be considered lyrics.

Perfect: If the query intent is very clear, the song matching that lyrics query.
1. [you used to call me on my cell phone] — this points to the popular song Hotline Bling by Drake
2. [you used to call me on my cell phone drake] — since the query is still mainly referring to the content (song Hotline Bling by Drake) by lyrics, it should be classified as Lyrics
3. [every night in my dreams i see you titanic] — the query refers to the song My Heart Will Go On by Celine Dion from the movie Titanic; even though it uses the lyrics with the name of the movie, it should be classified as
Lyrics, not Soundtrack
4. [i’m never gonna dance again] — points to the popular song Careless Whisper by George Michael

Excellent: More than one likely intents that are equally popular and relevant.
Songs when the intent in not clear and there are multiple equally popular songs to satisfy the intent (there's no clear separation between primary vs secondary intent).
Music video of the primary intent song implied by the lyrics.

1. [who do you think you are] — Jar of Hearts by Christina Perry

2. [who do you think you are] — the song by Spice Girls

Good: A popular piece of content which only partially satisfies the intent of the song query (content related to primary intent or popular secondary intents).
Popular remixes and covers of the intended song implied by the lyrics
Artist’s page of the intended song implied by the lyrics
Album of the intended song implied by the lyrics
Artist Essentials playlists containing the primary intent song implied by the lyrics
Other music of the artist
High-quality playlists that feature the intended song AND songs by other artists in similar genre (note that low-quality is Acceptable)
Popular songs that satisfy clear secondary intent

1. [i’m never gonna dance again] — Careless Whisper by Seether (a popular cover)
2. [you used to call me on my cell phone] —the artist page for Drake

Acceptable: a piece of content that poorly satisfies a secondary intent of the query, such as unpopular/unknown songs containing the queried for lyrics, or unpopular title/artist name keyword matches.
1. [i lean not on my own understanding] — Nothing I Hold Onto by Tracy Nitschke. Unpopular song by an unknown artist (around 500 views on Youtube) containing the queried for lyrics.

Unacceptable (Off-Topic): A piece of content that does not satisfy any of the above conditions.

Genre/Category
Some music queries can be classified as genre or category. These queries are not for a specific piece of content, but usually for a collection of content with a specific theme. Examples are queries like “songs for partying”,
“sad music”,“heavy metal”, “music for a weekend bbq”.

Perfect: Apple Music curated Playlists, Stations, Category Pages that contain exclusively songs matching the exact intent of the query. Please note that a playlist needs to match all elements of the query, and the title of
the playlist needs to thematically match the intent of a query. For example "slow jazz playlist" would not be Perfect for a query like "slow jazz for dinner" since the element of dinner is missing.
1. [pop] — Apple Music Pop
2. [pop] — The A-List: Pop playlist by Apple Music
3. [pop] — Apple Music pop Category page
4. [white noise] — White Noise 1 Hour Long
5. [hits] — Apple Music's charts Top 100: Italy (if the query is in Italian storefront)
6. [90s music] — Apple Music Category page for 90s music

https://baseline.apple.com/training/evaluations/1048/guidelines 7/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

Excellent: Third Party curated Playlists/Stations/Albums that contain exclusively songs matching the exact intent of the query as well as Apple Music Playlist/Stations/Albums that are more specific than the intent.
1. [pop] – Apple Music playlist Pop Hits: 2020 (playlists in the intended genre that is a little specific on era/year)
2. [pop] – Apple Music playlist Workout Pop (playlists in the intended genre that is a little specific on activity)
3. [pop] – Apple Music playlist Heartbreak Pop (playlists in the intended genre that is a little specific on mood)
4. [hits] – Top 100: Global
5. [90s music] – Apple Music playlist '90s Alternative Essentials
6. [pop] – NOW That’s What I Call Music! Vol. 73 (a popular album by various artists)
Good: Popular pieces of content which only partially satisfies the intent of the genre.
Popular Songs, single-artist Albums and Music Videos that match the exact intent of the query
Artist pages if the query limits the intent to less than 3 artists
Artist Essentials playlists if all songs fall into the requested category
Playlists that only matches the intent partially and are less specific than the intent
1. [rock music] - Mr. Brightside by The Killers
2. [music from the 90s] - No Scrubs by TLC
3. [songs from the 70s from the beatles] - The Beatles artist page
4. [90s workout pop] - 90s pop playlist
Acceptable: Pieces of content that poorly satisfies a secondary intent of the query.
Unpopular Songs, single-artist Albums and Music Videos that match the exact intent of the query
Unpopular matching titles that can poorly satisfy the intent of the query
1. [dance music] - Lit Cigarette by SDC
2. [beach music] - Beach Music by J.D. Pederson

Unacceptable (Off-Topic): Any other content

1. [rock] — a piano concerto by Beethoven
2. [rock] — the artist Bob Marley
3. [hits] — non-Apple Music's playlist Ballermann Hits 2022 (if the query is in Italian storefront)
4. [90s music] — Apple Music playlist Hip-Hop Hits: 2020

Ambiguous

A query that is too ambiguous to identify a single primary intent. These queries may fit into several intent categories; for example the query [blue] could refer to a mood, a song, or an artist called Blue. You should decide
on the most likely classification for the query and use the rating for that classification accordingly. Describe your reasoning in the comments.
Let's take query [blue] as an example. If it is considered a genre, a high quality playlist can be rated Perfect. If there is a very popular song with that title, it can be rated Perfect as well. If there is a popular artist with the
name Blue and you believe it would be a primary intent, the song by that artist would be rated Excellent.

Perfect: A piece of content which is very likely to satisfy a primary intent of the ambiguous query. Only the most popular content will be eligible for this rating

[blue] – Apple Category page Feeling Blue (if we consider the query to be a genre query, the category page is rated Perfect)
[blue] – Song Blue (Da Ba Dee) by Eiffel 65 (if we consider the query to be a song query, the most popular song that satisfies primary intent is considered Perfect)

Excellent: A piece of content which clearly satisfies the intent of the query but is less popular or less known.

Good: A piece of content which only partially satisfies the intent of the query, but is still high-quality OR satisfies a secondary intent of the query OR satisfies the intent of the query but is lower quality.
[blue] – artist Blue October (if we consider the query to be an artist query, this result is relevant through text match but not that popular, so would only satisfy secondary intent)

Acceptable: A piece of content that is related to the query, but may be of poor quality, not popular, or poorly satisfies a secondary intent of the query.

Unacceptable (Off-Topic): A piece of content that does not satisfy any of the above conditions.

Curator

Perfect -- the intended Curator page

1. [Spinnin’ Records] -- the curator page for Spinning Records
2. [boiler room] -- the curator page for Boiler Room
3. [tiktok] -- the curator page for TikTok

Excellent -- curated playlist, and albums by the intended Curator (regardless of popularity) are all considered Excellent results. Also, relevant and high quality playlists from other reputable providers can receive an
Excellent rating.
1. [Spinnin’ Records] -- Hardest Drops - by Spinnin' Records
2. [Spinnin’ Records] -- Compilation album by the curator Spinnin Records Best of Dance 2020
3. [boiler room] -- Boiler Room Radio: Extended Interviews - Pierre Kwenders
4. [boiler room] -- Boiler Room Playlist - Ep 3
5. [tiktok] -- TikTok Dance playlist by the intended curator
6. [tiktok] -- TikTok Songs playlist by Topsify. This is a highly relevant playlist from a well know curator, featuring the biggest TikTok viral hits.

Good — a piece of content which only partially satisfies the intent of the curator query such as Songs on a curated compilation album.
1. [Spinnin’ Records] -- Song on a compilation album by the Curator Day Or Night
2. [Spinnin’ Records] -- Song on a compilation album by the Curator Grapevine
3. [Spinnin’ Records] -- Song on a compilation album by the Curator Escape
4. [boiler room] -- Song on a compilation album by the Curator Y Is la Ves
5. [tiktok] -- Kesha's song Tik Tok. This song is from 2010 and no longer the primary intent of most users. Satisfies a secondary intent and is rated Good.

Acceptable -- a piece of content which only poorly satisfies the intent of the curator query. This may include curator Uploaded Content of lower quality and unpopular pieces of content with the same name or title (the
piece of content has no ratings, or the content has negative ratings), or matching part of the title.
1. [Spinnin’ Records] -- a song titled ‘Spinning Records‘
2. [Spinnin’ Records] -- an unknown song titled ‘Spinning Records‘
3. [tiktok] -- an unknown song by an artist titled Tik Tok

Unacceptable (Off-Topic) -- a piece of content that is not related to the Curator, has no perceived relevance, or is likely to mislead users about its authorship. This also includes pieces of content featuring similar names
or titles, and media for which the connection with the curator is not readily apparent.
1. [Spinnin’ Records] -- Song of an album that is not by the curator Issues (feat. Ice Cube, Anderson .Paak & Dem Jointz)
2. [Spinnin’ Records] -- Song of an album that is not by the curator I'm the One (feat. Justin Bieber, Quavo, Chance the Rapper & Lil Wayne)
3. [boiler room] -- Song Boiler by Limp Bizkit
4. [tiktok] -- an artist named T.O.K

Record Label

Perfect -- The particular record label implied by the query.

1. [fat possum] — the record label page for Fat Possum Records
2. [b’in music] — the record label page for 相信音樂 (B’in Music in English)
3. [glassnote] — the record label page for Glassnote Records
4. [dead oceans] — the record label page for Dead Oceans

Excellent — Record label’s Curator’s page, and all content either curated, or produced by the intended record label. This includes Artist pages for artists signed by the record label, Playlists, Albums, Songs, and Radio
Shows produced, and/or curated by the record label.
1. [fat possum] — the curator page for Fat Possum Records
2. [dead oceans] — the curator page for Dead Oceans
3. [fat possum] — the playlist Fat Possum: Album of the Week by Fat Possum Records
4. [drag city] — the playlist Drag City For The Rest of Us by Drag City
5. [mau5trap] — the radio show mau5trap presents...
6. [604 records] — the album A 604 Records Christmas by Various Artists (album produced by 604 Records)
7. [604 records] — the song Mittens (Robert Orton Mix) by Carly Rae Jepsen, and other songs from the album 604 Records: Bundled Up produced by 604 Records

Good — a popular piece of content which only partially satisfies the intent of the query.
1. [drag city] — the song Drag City by Jan & Dean
2. [heavenly sweetness] — the EP album Heavenly Sweetness Loves Calypso Rose by Calypso Rose (this album isn’t produced by the intended record label)

Acceptable -- a piece of content that is related to the query, but is unpopular or only poorly satisfies a secondary intent. This includes unpopular pieces of content matching, or partially matching the query; and content
related to the query.
1. [fat possum] — the song Fat Possum by Parry Music. This is not a popular song; and it’s not produced by the record label
2. [fat possum] — the artist page for Parry Music. The artist has a song called ‘Fat Possum’; and he’s not produced by the record label
3. [dead oceans] — the artist page for the unpopular artist Dead Oceans
4. [dead oceans] — the song Don’t Wait by unpopular artist Dead Oceans
5. [drag city] — the artist page for Jean & Dean
6. [drag city] — the song Drag City by Baga chipz Mbe (unpopular song)

Unacceptable (Off-Topic) -- a piece of content that is not related to the record label. , and content that is irrelevant to the record label.
1. [fat possum] — the album Bluegrass Trash by Fat Man Squeeze. This album is not relevant to the query, and is not produced by the record label
2. [fat possum] — the album Let Music & Bodies Unite by Rat Vs Possum. This album is not relevant to the query, and is not produced by the record label

https://baseline.apple.com/training/evaluations/1048/guidelines 8/9
19:33 4/9/24 Guidelines for Search - Music v4 Training — BaseLine

3. [dead oceans] — the playlist Dead Can Dance Essentials. The artist Dead Can Dance is not produced by the intended record label, and the band is not relevant to the query

Video Navigational

A query that refers to a specific Apple Music tv show or episode, as well as music videos for a specific song. Some query examples are [carpool karaoke], [planet of the apps], [sicko mode video], etc.

Perfect – Results that satisfy primary intent of the query.

The intended tv show container/episode
Specific song's music video implied by the query
Compilation of artist's videos implied by the query
1. [carpool karaoke] – Carpool Karaoke The Series
2. [sicko mode video] – The music video for the song Sicko Mode by Travis Scott
3. [carpool karaoke shakira] – The Carpool Karaoke episode featuring Shakira
4. [justin bieber music videos] – Justin Bieber Video Essentials
5. [peaches video] – Video Peaches by Justin Bieber

Excellent – A piece of content that satisfies a dominant intent of the query and is relevant to the query.
The intended music video’s song
Individual songs by the queried artist.
Single episodes from the intended Apple Music TV show when the intent is for the full show, and vice versa.
1. [sicko mode video] – The song Sicko Mode by Travis Scott
2. [toosie slide music video] – The song Toosie Slide by Drake
3. [carpool karaoke] – Carpool Karaoke Shakira & Trevor Noah episode
4. [carpool karaoke shakira] – Carpool Karaoke The Series
5. [justin bieber music videos] – Individual song's video Peaches by Justin Bieber
6. [peaches video] – Song Peaches by Justin Bieber (the intended music video’s song)

Good – A piece of content that satisfies a secondary intent.

Artist page for the intended music video’s artist
Other content by the artist: songs, albums, and music videos from the intended artist.
Video of songs that could satisfy secondary intent of the query
1. [sicko mode video] – The music video for the song Yosemite by Travis Scott - Another song/video by the intended artist
2. [sicko mode video] – The artist page for Travis Scott
3. [justin bieber music videos] – Artist page for Justin Bieber
4. [peaches video] – Album Justice for Justin Bieber (album containing the song)
5. [peaches video] – Video of song Peaches by KAI (video of songs that could satisfy secondary intent of the query)
6. [peaches video] – Video of song Anyone by Justin Bieber (other content by the artist)

Acceptable — A piece of content which only poorly satisfies the intent of the video query.
This includes content of lower quality and unpopular pieces of content with the same name or title.
Playlists with the intended song/content
1. [sicko mode video] — Sicko Mode [B-Mix] music video by Bagstheboss (this is an unpopular cover of the intended song)
2. [peaches video] – Playlist Pop Drive (playlist with the queried song and others in similar genre)

Unacceptable (Off-Topic) – a piece of content that is not related to the query, has no perceived relevance, or is likely to mislead users about its authorship.

Copyright © 2024 Apple Inc. All rights reserved. Apple Confidential.

https://baseline.apple.com/training/evaluations/1048/guidelines 9/9