Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

Ad Delivery Algorithms:
The Hidden Arbiters of Political Messaging
Muhammad Ali∗ Piotr Sapiezynski∗ Aleksandra Korolova
mali@ccs.neu.edu p.sapiezynski@northeastern.edu korolova@usc.edu
Northeastern University Northeastern University University of Southern California
Boston, MA, USA Boston, MA, USA Los Angeles, CA, USA

Alan Mislove Aaron Rieke

amislove@ccs.neu.edu aaron@upturn.org
Northeastern University Upturn
Boston, MA, USA Washington, DC, USA

ABSTRACT ACM Reference Format:

Political campaigns are increasingly turning to targeted advertising Muhammad Ali, Piotr Sapiezynski, Aleksandra Korolova, Alan Mislove,
and Aaron Rieke. 2021. Ad Delivery Algorithms: The Hidden Arbiters of
platforms to inform and mobilize potential voters. The appeal of
Political Messaging. In Proceedings of the Fourteenth ACM International
these platforms stems from their promise to empower advertisers Conference on Web Search and Data Mining (WSDM ’21), March 8–12, 2021,
to select (or “target") users who see their messages with great pre- Virtual Event, Israel. ACM, New York, NY, USA, 9 pages. https://doi.org/10.
cision, including through inferences about those users’ interests 1145/3437963.3441801
and political affiliations. However, prior work has shown that the
targeting may not work as intended, as platforms’ ad delivery al-
gorithms play a crucial role in selecting which subgroups of the
targeted users see the ads. In particular, the platforms can selec- 1 INTRODUCTION
tively deliver ads to subgroups within the target audiences selected Political campaigns spend millions of dollars on advertising to get
by advertisers in ways that can lead to demographic skews along their messages out to voters. This spending has been increasingly
race and gender lines, and do so without the advertiser’s knowl- migrating from traditional broadcast media (e.g., television and
edge. In this work we demonstrate that ad delivery algorithms used newspapers) to the internet. For example, at the U.S. state level, ten
by Facebook, the most advanced targeted advertising platform, times as many candidates advertise on Facebook than on TV [13].
shape the political ad delivery in ways that may not be beneficial The reason for the shift is that online advertising platforms promise
to the political campaigns and to societal discourse. In particular, to lower the cost of advertising and increase the efficacy of cam-
the ad delivery algorithms lead to political messages on Facebook paigns through detailed targeting, where advertisers can specify the
being shown predominantly to people who Facebook thinks al- users they would like to reach using attributes. Common targeting
ready agree with the ad campaign’s message even if the political attributes include demographic characteristics, interests, visited lo-
advertiser targets an ideologically diverse audience. Furthermore, cations, phone numbers, relationship status, and wealth. Platforms
an advertiser determined to reach ideologically non-aligned users is are known to determine attributes in a variety of ways, including
non-transparently charged a high premium compared to their more by relying on those provided by the users explicitly, algorithmic
aligned competitor, a difference from traditional broadcast media. inferences by the platforms based on users’ activity (both on the
Our results demonstrate that Facebook exercises control over who platforms and elsewhere), or augmentation with data supplied by
sees which political messages beyond the control of those who pay third-party data brokers [8, 10, 12, 32].
for them or those who are exposed to them. Taken together, our The implications of the ad creation and targeting phase of the
findings suggest that the political discourse’s increased reliance on advertising process, where the advertiser uploads their ad creative
profit-optimized, non-transparent algorithmic systems comes at a and selects their desired target users, have been subject to recent
cost of diversity of political views that voters are exposed to. Thus, scrutiny and policy debate. Targeting attributes offered by the ad
the work raises important questions of fairness and accountability platforms have been shown to enable advertisers to prevent certain
desiderata for ad delivery algorithms applied to political ads. ethnic groups from seeing ads [5, 29]. For example, in 2016, the
Trump campaign used these techniques in an attempt to lower
∗ These authors contributed equally to this research. turnout among young women and Black voters [17], and there is
evidence that Russian organizations used these tools with the goal
of influencing the 2016 U.S. presidential elections [27, 30]. Recently,
U.S. Federal Election Commission (FEC) Chair Ellen Weintraub
argued that platforms should limit political advertisers’ ability to
This work is licensed under a Creative Commons Attribution-NonCommercial- narrowly target ads to ensure that “a broad public can hear the
NoDerivs International 4.0 License.
speech and respond” [33]. Shortly after, Google announced that it
would significantly limit election ad targeting in order to “promote
WSDM ’21, March 8–12, 2021, Virtual Event, Israel increased visibility of election ads” [16].
© 2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8297-7/21/03.
https://doi.org/10.1145/3437963.3441801

13
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

Scientific attempts to rigorously measure the effects of online regards to political message delivery to the platforms; and there-
political targeted advertising outside of the controlled lab environ- fore, makes the investigation of their delivery algorithms and their
ments [22] have been limited by the challenge of controlling for the implications for political discourse even more important.
optimization decisions made by platforms in their ad delivery phase, To rigorously answer the questions posed, we became a polit-
or the process by which platforms select which ads get shown to ical advertiser and spent over $13,0001 to run political ads under
which users [9]. controlled conditions, and observed how Facebook’s algorithms de-
Given the platforms’ desire not to overwhelm the users with livered them. We used Facebook’s ad reporting features, combined
too many ads (especially those with potentially upsetting content), with proxies, to understand who our ads were delivered to and how
the finite budget of the advertiser, a large potential audience of ad our budget was split across users with different political leanings.
recipients, and the competition from other advertisers for those
recipients, platforms need to select a subset of the targeted users 1.1 Main Findings
who will actually be shown the ad. This selection is commonly The results of our analyses offer the following contributions:
performed through auctions, where the outcome is determined
First, we show that, despite identical targeting parameters, budgets,
not only based on the advertisers’ willingness to pay, but also on
and competition from other advertisers, the content of a politi-
the platform’s long-term business and growth goals, such as the
cal ad alone can significantly affect which users Facebook will
platform’s desire for its users to see relevant ads (and, therefore,
show the ad to. For example, when we run two campaigns, each
maintain its user base) and the platform’s desire for its advertis-
targeting the same audience comprised of an equal number of reg-
ers to achieve their desired outcomes (and, therefore, achieve the
istered Democratic voters and Republican voters, we find that our
platform’s revenue and advertiser growth goals). We call the al-
ad for a Democratic candidate delivers to an audience that is 70%
gorithmic approach that the platform uses to balance these goals
Democratic, while the audience reached by the ad for a Republican
during the selection ad delivery optimization.
candidate is only 40% Democratic.
Prior work has showed that ad delivery optimization, and its re-
liance on algorithmically inferred “relevance" of an ad to a user, can Second, we find that it can be difficult and more expensive for
lead to troubling results in the context of life-opportunity ads [4]. political campaigns to have their content delivered to those who
Specifically, it demonstrated that ads targeting the same gender- Facebook believes are not aligned with the campaign’s views. For
and race-balanced audiences for various jobs were delivered to example, we find that when targeting an audience of conserva-
vastly different groups of users: cashier job openings were shown tive users, in the first day of the ad campaign, Facebook delivers
predominantly to women, taxi driver job openings were shown our liberal-leaning ad to only 4,772 users, while our conservative-
predominantly to Black users, and artificial intelligence and lumber leaning ad to 7,588 users.2 We find that the underlying reason for
jobs openings were shown to majority white and male audiences [4]. the differences in delivery is that our liberal-leaning ads target-
These results were subsequently extended with controls for user ing conservative users are charged significantly more by Facebook
job qualifications [18] and reproduced in several European coun- than our conservative-leaning ads ($15.39 versus $10.98 for 1,000
tries [19]. Further prior work showed that the existing ad ecosystem impressions), despite being run from the same ad account, at the
provides little support for advertising to a demographically repre- same time, and targeting the same users.3
sentative cohort [14]. Third, we observe these effects persist for ads that do not prompt
In this work, we investigate the impact of ad delivery optimiza- user engagement, which suggests that the ad delivery decisions
tion in real-world advertising platforms on a different arena: po- made by Facebook are not driven exclusively by user reactions to
litical discourse. We focus on Facebook because of its critical im- the ad but instead are made at least partially by Facebook itself.
portance to today’s digital political advertising and its pioneering Taken together, our results indicate that Facebook preferentially
role in targeted advertising. Specifically, we seek to answer: Is a shows users political ads whose contents Facebook predicts are
political campaign advertising on Facebook able to reach all of the aligned with their political views. This ad delivery choice has neg-
electorate? Or, is Facebook preferentially delivering ads to users who ative implications for both users and campaigns. For users, such
it believes are more likely to be aligned with the campaign’s political delivery limits users’ exposure to diverse viewpoints unbeknownst
views? Additionally, to what extent does Facebook vary ad pricing to the users, especially if the predictions about the alignment are
based on its hypothesized match between the target audience’s and based on an algorithmic analysis of users’ activity and third-party
campaign’s political views? data, rather than information explicitly provided by the users. For
The answers to these questions are particularly urgent and campaigns, such delivery may inhibit them from reaching beyond
salient in light of the debate unfolding over the “microtargeting” of their existing “base” on Facebook, as getting ads delivered to users
political ads for at least two reasons. First, skews resulting from ad the platform believes are not aligned with their views may become
delivery can raise similar concerns to those raised about narrow prohibitively expensive. Furthermore, unlike in traditional media,
targeting: an electorate who cannot “hear and respond” to political this may imply that campaigns of equal financial means are not
speech. Second, ad delivery algorithms might counteract the goals equal in their ability to reach a particular audience4 , with the price
of restricting microtargeting by redirecting ads according to the differential decided exclusively and non-transparently by Facebook.
choices of the platforms (in spite of broader target audiences). In
1 Throughout the paper we refer to prices in U.S. Dollars.
other words, limiting targeting options transfers more power with 2 We find a similar, but flipped, effect if we target an audience of liberal users.
3 Again, we see a similar, flipped effect when targeting liberal users.
4 The work of [23] hypothesizes differential pricing using a different methodology.

14
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

Importantly, these effects may be occurring without users’ or cam- engagement with US political content (Liberal)”. We further nar-
paigns’ knowledge or control. rowed the targeting by specifying additional required characteris-
Stepping back, our findings raise serious concerns about whether tics such as those who are, according to Facebook’s characteriza-
Facebook and similar ad targeting platforms are, in fact, amplifying tion, “interested in” topics such as “Donald Trump for President”,
political filter bubbles by economically dis-incentivizing content “Make America Great Again”, or “Bernie Sanders”. We aimed to
they predict is not aligned with users’ political views. Put simply, approximately match the sizes of liberal- and conservative-leaning
Facebook is making decisions about which political ads to show to audiences for the region by adjusting the targeting radius around
which users based on its own priorities, such as user engagement a chosen location until the Estimated Daily Reach provided by
or financial growth. Although Facebook’s role was not entirely Facebook was close to matching.
unpredictable given the previous work on delivery optimization in
the context of job ads [4], we confirm it extends to political adver-
tising, a context in which Facebook’s choices may have significant 2.2 Ad copy creation
negative externalities on political discourse in society at large. We registered as political advertisers on Facebook (which required
Our investigation presents a new example of an empirical study confirming our identity and residence in the United States). In our
of a black-box algorithmic system, and the challenges of pursuing experiments we ran two types of ads: generic and real.
such a study. It thus raises questions of the accountability desiderata The generic ads did not feature any candidate or a political
for ad delivery optimization in the context of political advertising. stance. Instead, they showed an image of the American flag, and the
ad copy encouraged the viewers to register to vote, see Figure 1e.
Ethics All of our experiments were conducted with careful con- All ads appeared to point to our domain psdigital.info, but if
sideration of ethics. First, we obtained Institutional Review Board any user clicked the ad, they would be redirected to the official
review of our study, with our protocol being marked as “Exempt”. fec.gov webpage.
We did not collect any users’ personally identifying information The majority of the real ads replicated the ads run by official
from Facebook, and did not collect any information about users political campaigns that we obtained from the Facebook Ad Li-
who visited our site after clicking on our ads. Second, we minimized brary [11], see Figure 1a-d. Ads for Bernie Sanders’ merchandise
harm to Facebook users when running our ads by only running store were the only exception, as his campaign had not advertised
“real” ads, i.e., if a user clicked on one of our ads, they were brought merchandise on Facebook; we created the ad creative for this ad.
to a real-world page not under our control that was relevant to Whenever the replicated ad was written in the first person, we
the topic of the ad. In the few cases where the ads pointed to a changed it to be a third person reference to the name of the can-
domain we controlled, the visiting users were automatically and didate. We chose Donald Trump and Bernie Sanders for the ads
immediately redirected to a real page that we did not control. Third, because at the time of experiment design (early July 2019), they had
we minimized harm to Facebook itself by participating in their ad- spent most on Facebook advertising among the major candidates
vertising system as any other advertiser would and paying for all of of each party [11].
our ads. We registered as an advertiser in the area of “Social Issues, Each of our experiments consisted of two ad campaigns: one
Elections or Politics” [2], meaning our ads were subject to the same with a copy and/or linked content that is liberal-leaning and the
review as the ads of other political campaigns. Fourth, we minimized other – that is conservative-leaning. Each of these ad campaigns,
the risk of altering the political discourse through careful choices in turn, featured two ads that looked exactly the same to the users,
of the ad content, and running approximately the same number of but targeted different audiences: one targeting a liberal-leaning
copies of ads for Republican and Democratic candidates, with the audience, and another – a conservative-leaning one. Therefore,
same budgets. The total amount we spent on political advertising each experiment consists of four ads in total.
while collecting data for this paper was minuscule compared to It is worth noting that using two candidates/parties is sufficient
the ad budgets of real campaigns in the same period (likely in the for studying political ads in the U.S. because of the primarily two-
millions of dollars [11]). party system (Republicans vs. Democrats). Our proposed method-
ology can be extended to multiple political parties to understand
skews in other countries where several parties might be competing
2 METHODOLOGY in an election.
Our experiments consist of four stages: audience creation, ad cre-
ation, collection of data on delivery, and statistical analysis. Here,
2.3 Performance optimization and statistics
we briefly describe the decisions made at each stage.
When creating ad campaigns, advertisers on Facebook are asked
to specify their objective, or what they are trying to achieve, and
the optimization that Facebook should use to achieve the objec-
2.1 Audience creation tive. Unless stated otherwise, all of our campaigns ran with the
Facebook’s advertising interface allows us to target users based “Reach" objective and “Reach" optimization, which according to
on their inferred political interests. We created two audiences this Facebook’s documentation [26], means Facebook would allocate
way, selecting a geographic region centered around a town and the campaign’s specified budget to maximize the number of unique
Facebook’s inferred characterization of interests such as “Likely users to whom the ad is shown, rather than to maximize, for ex-
engagement with US political content (Conservative)” and “Likely ample, engagement or “Traffic" (showing the ad to the users most

15
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

a) Trump - Issues b) Trump - Merch c) Sanders - Issues d) Sanders - Merch e) Generic

e) Neutral

Figure 1: Ads used in our experiments concerning political issues and promoting candidates’ merchandise.

likely to click). Consequently, our campaigns were charged for im- Sanders Trump
pressions rather than for clicks. Although we cannot verify this A B
as Facebook’s political ad archive does not reveal the optimization Sanders
objectives of political ad campaigns, we hypothesize that political
Trump
campaigns aiming to get their message out to as many people as
possible and / or aiming to reach an ideologically diverse audience 0.40 0.45 0.50 1.0 1.2 1.4 1.6
are likely to use this combination of objective and optimization. Estimated fraction Cost penalty
After choosing the audiences and selecting the ad copies, we of Democrats in the audience for non-alignment
submitted our campaigns for review. Upon acceptance, the ad cam-
paigns started presenting the ads to users. Using the advertising Figure 2: Delivery statistics for ads that look identical to
interface, we tracked the number of users reached by each ad, as users, but appear partisan to the Facebook classification
well as the cost Facebook charged us for impressions every five mechanism. (A) Ads that appear to promote the Democratic
minutes over the entire lifetime of the ads. candidate are shown more to liberal users and vice-versa. (B)
The financial penalty for trying to show an ad that Facebook
2.4 Statistical analysis deems non-aligned; reaching the same number of people in
In the course of this work we compare the fractions of Democrats the same audience is up to 1.4 times more expensive.
(or Republicans) among the users exposed to two ads that differ in
their content. The comparison process consists of two steps and is
based on previous work [4]. visually from the subsequent figures), the difference is statistically
First, we estimate the fraction of Democrats in each ad, and the significant.
99% confidence interval around that estimate using the method
recommended by Agresti and Coull [3], shown in Equation (1): 3 RESULTS
r We study what happens when a political campaign places ads on
𝑧2
𝛼 /2 𝑝ˆ (1−𝑝)
ˆ 𝑧2
𝛼 /2
𝑝ˆ + 2𝑛 − 𝑧𝛼/2 + 4𝑛 2
Facebook to an audience (the set of targeted users) containing both
𝑛
𝐿.𝐿. = 2 /𝑛
, users who likely agree with the campaign’s views (e.g., to solicit
1 + 𝑧𝛼/2 donations, or to increase engagement) as well as users who likely
r (1)
𝑧𝛼2 /2 𝑧𝛼2 /2
disagree with the campaign’s views (e.g., to try and change their
𝑝ˆ (1−𝑝)
ˆ
𝑝ˆ + 2𝑛 + 𝑧𝛼/2 𝑛 + 4𝑛2 minds). We do so first by exploring the impact of the ad platform’s
𝑈 .𝐿. = 2 /𝑛
, relevance estimates on generic ads, and then demonstrate the im-
1 + 𝑧𝛼/2
pact on real-world ads by running ads similar to those of actual
where 𝐿.𝐿. is the lower confidence limit, 𝑈 .𝐿. is the upper confi- campaigns.
dence limit, 𝑝ˆ is the observed fraction of Democrats in the audience,
𝑛 is the total size of the audience exposed to the ad. To obtain the 3.1 Generic ads
99% interval we set 𝑧𝛼/2 = 2.576. We first aim to isolate the impact of the ad platform’s relevance
Second, we compare whether the fractions in two scenarios are estimates, and to avoid any interference from the reactions of the
statistically significantly different. Since in the vast majority of users themselves, which can be different across groups with dif-
our results the confidence intervals do not overlap (easily judged ferent political leanings. To do so, we run several copies of ads

16
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

that appear identical to users (Figure 1e) but differ in the political each user each week, thus preventing the delivery mechanism from
leaning of the landing page, i.e. the webpage that the ad links to. showing the ad to the same subset of users repeatedly. Given our
To achieve that, we configure our servers so that when viewed by audience size and budget, we expected to reach almost everybody
a real user, the landing page of all ads is a U.S. government web- in the audience by the end of the run.
site with instructions on how to register to vote. However, when The results of this experiment are presented in Figure 3. We
visited by Facebook’s web crawler5 , each ad’s landing page shows first focus on panel A, which shows the cumulative number of
different content: one serves Trump’s campaign content, another users reached over seven days. We can observe two notable effects:
serves Sanders’. Since all ads look entirely identical to users, any the smaller reach of the Trump ad targeting the liberal-leaning
skew in delivery can only be attributed to Facebook’s relevance audience and the Sanders ad targeting the conservative-leaning
estimates based on the content of the linked website. audience. Both of these non-aligned ads end up delivering to over
We set a budget of $40 per day for each of our two ad campaigns 20% fewer users than their aligned counterparts. Specifically, when
and run them simultaneously from the same advertising account the Trump ad is targeted to the conservative audience, it delivers to
for two days. We run two identical copies of each ad: one targeting a total of 21,792 users; when the Sanders ad is run at the same time
a liberal-leaning audience of 3,000 users in a single city, and the and targeted to the same conservative audience, it delivers to only
other – a conservative-leaning audience of 3,600 users in the same 17,964 users. This difference in reach cannot be attributed to an
city. underlying difference in users’ likelihood to click on the ads, as we
The two audiences are disjoint, so for each ad, we divide the configured the campaigns to pay per ad impression and optimize
number of users reached in the ad targeting the liberal-leaning for reach, not clicks.
audience by the total number of users reached in both copies; the Figure 3B shows that despite equal budgets for all ads, the
results are presented in Figure 2A. Even though the users see the Sanders campaign targeting liberal users slowed down the spending
same ad in both cases—meaning users’ explicit or implicit reactions after the second day and, as an effect, spent less than $450 of the
are no more different than chance—we observe that delivery is allocated $700. In fact, after this point, the campaign did not reach
skewed according to the political leaning of the landing page (with many more users, as seen in panel A.
the ad with Sanders’ landing page being delivered to the highest We turn to panel C, which shows the cumulative cost per 1,000
fraction of liberal-leaning users, and Trump’s, the lowest). We also unique users to help explain why this effect is occurring. We can
calculate the cost penalty by comparing prices of reaching the first immediately notice an increasing cost trend for all ads: as the ads
1,000 users in each audience for the non-aligned ad versus the run longer, their cost per 1,000 reached users increases substantially.
aligned ad; the results are presented in Figure 2B. We observe that Presumably, this is because Facebook first delivers the ad to the
it costs 1.4 times more for the ad with Sanders’ landing page (as “cheaper” users in the target audience before deciding to spend our
perceived by Facebook) to reach the same number of users in a budget on the more “expensive" users (recall, we prevented Face-
conservative audience than for the ad with Trump’s landing page. book from delivering ads to users more than once). However, we
Conversely, it costs 1.4× more for the ad with Trump’s landing page can observe that the non-aligned ads are again outliers: both show
to reach the same number of users in a liberal audience than for a substantially higher cost per 1,000 users, a difference noticeable
the ad with Sanders’ landing page. from the start of the experiment. By the end, when the liberal ad is
These results show that the content of the landing page—and delivered to the liberal-leaning audience, it is charged $21 per 1,000
not only users’ reaction or engagement with the ad, or the compe- users; when the conservative ad is delivered to the same audience,
tition from other advertisers—plays a significant role in Facebook’s it is charged over $40 per 1,000 users.
ad delivery optimization decisions, and can result in both skewed Because the delivery rates slow down after the first day, Fig-
delivery and differential pricing, despite inclusive targeting by the ure 3C makes the growth of cost per 1,000 also appear to slow
campaign. As a result, two political campaigns running ads con- down. Therefore, we turn to Figure 3D which shows this growth
cerning the same issue to the same target audience may reach as a function of the size of reached audience, rather than time. We
different sub-populations of that audience and at different prices, observe that the growth is rapid and accelerating, especially for
only because their landing pages are different. non-aligned ads. Finally, in Figure 3E we show that the ratio be-
tween the cost a political campaign pays to show their ad to the
3.2 Real ads non-aligned audience and the cost of their competitor showing to
We now explore the implications of ad delivery optimization for the same audience is relatively stable, between 2:1 and 3.5:1.
real-world ads that differ both in the ad content and landing page. In Overall, Figure 3 emphasizes three findings relevant to the ques-
this experiment, we run two ads (again, one for Trump, and one for tions we posed: First, by the end of the experiment, the two aligned
Sanders, using ad creatives a and c from Figure 1 respectively), each partisan ads reached over 20,000 users, while the non-aligned ads
to two audiences of over 30,000 liberal- and conservative-leaning reached many fewer–Facebook limited the delivery of ads whose
users6 over a period of seven days in August 2019 and with a daily content did not agree with the audience’s inferred political leaning
budget of $100 for each ad and audience combination. For these (Figure 3A). Second, among two campaigns trying to reach the same
ads, we specify that we only want to show the ad at most once to audience, the one that Facebook deems non-aligned will pay a sig-
nificant cost penalty (see Figure 3E). Third, while the cost per 1,000
5 We determined Facebook IP addresses by using the IP address blocks advertised by
reached users grows with time in a sub-linear fashion (Figure 3C),
Autonomous Systems numbers owned by Facebook.
6 Using Facebook’s interest based targeting “Likely engagement with US political it grows super-linearly with the number of users already reached
content (Conservative)” and “Likely engagement with US political content (Liberal)” (Figure 3D).

17
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

Sanders; liberal audience Sanders; conservative audience Trump; conservative audience Trump; liberal audience
800
A B 40 C 40 D E
20k

1000 users [USD]

1000 users [USD]

600 3

Spend [USD]

Cost penalty
Cost per

Cost per
Reach

400
10k 20 20 2
200
Equal cost
0 0 0 0 1
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 5k 10k 15k 20k 0 5k 10k 15k 20k
Campaign duration [days] Campaign duration [days] Campaign duration [days] Users reached Users reached

Figure 3: Ads for a political campaign deliver to more users and for a lower cost if the targeted users have the same inferred
partisanship. A - the delivery rates are the highest in the beginning of the ad runtime and for aligned audiences. B - initially
all ads spend $100 per day, but the Sanders ad targeting liberal audiences does not spend its full budget. C - the cost of reaching
non-aligned audiences is higher. D - the more people have already seen the ad, the more expensive it becomes to show it to even
more people and the effect is even stronger for non-aligned audiences. E - the ratio between the cost of a political campaign
advertising to a non-aligned audience and their competitor advertising to the same audience.

Taken together, our findings demonstrate the core phenomenon: instead of voter records to construct custom audiences in the same
there’s a reach and cost penalty on those campaigns whose po- way (see the bottom row of Figure 4). The absence of a skew for
litical affiliation (or even merely the landing page) is (as inferred the donor record audiences might suggest that Facebook does not
by Facebook) not politically aligned with the target audience, as have sufficient information about these users.
compared to the campaigns targeting the same audience whose Next, we perform a series of additional experiments to verify
political affiliation is inferred to be aligned. the robustness of the results to the specific ad copy, audience size,
audience geographical location, and the level of detailed targeting.
3.3 Robustness of results The results are presented in Figure 5, with each experiment in a
separate row. We vary three aspects of our experiments:
In this section we describe our efforts to corroborate our findings
and show the robustness of the presented effects to changes in a (1) The size of the audience, as reported by Facebook’s Estimated
range of variables. Daily Reach,
First, we replicate the method that Ali et al. [4] used to measure (2) The “specificity” of the audience (narrowing the detailed tar-
racial skew in the delivery of job and housing ads. To that end, we geting further by attributes such as users’ inferred interest in
use the Custom Audiences CA𝐴 , CA𝐵 , CA𝐶 , and CA𝐷 described in “Donald Trump for President” or “Bernie Sanders” according
Table 1 that are based on publicly available voter records from North to Facebook), and
Carolina. These audiences are designed so that asking Facebook to (3) The specific topic of the ad (adding ads that advertise small
report delivery statistics by DMA serves as a proxy for obtaining campaign-branded merchandise that users can purchase, as
delivery statistics by political affiliation. shown in Figure 1).
We re-use ad creatives from the official Trump and Sanders We make a number of observations from this experiment. First,
Facebook pages (similar to Figure 1) and link to the respective we observe statistically significant skews in ad delivery along polit-
campaign’s web site. We then run one copy of each ad targeting ical lines for all of our ad configurations. This suggests that such
each of the four Custom Audiences, for a total of 8 individual ads. skew is a pervasive property of Facebook’s ad delivery system. Sec-
Our ads are run with a daily budget of $20 per ad set and use ond, we observe that the skews tend to be less pronounced when
the objective “Traffic” and optimization “Link Clicks” as in prior the ads are targeting larger audiences (more than 10,000 daily ac-
work [4]. tive users). While we do not know the underlying cause of this
Figure 4 (top row) presents the overall delivery statistics for these phenomenon, we hypothesize that the larger audiences provide
two ad creatives, with the delivery statistics of all four instances of the platform with a big enough pool of users to afford “relevant”
each ad aggregated together. We can immediately observe signifi- users regardless of their inferred political leaning. On the other
cant differences in delivery: the Trump ad delivers to less than 40% hand, we suspect that when running our ads with smaller audiences,
Democrats, while the Sanders ad delivers to almost 70% Democrats. Facebook “exhausts” the (small) subset of users in the non-aligned
Note that this difference in delivery is despite the fact that all ads are audience (e.g., Sanders advertising to a conservative audience) for
run from the same ad account, at the same time, targeting the same whom Facebook believes the ad is, in fact, relevant, and thus pauses
audiences, and using the same goal, bidding strategy, and budget; or raises the price for delivery, but continues the delivery among
the only difference between them is the content and destination link the aligned audience.
of the ad. This finding shows that the skewing effect persists even
if the advertiser does not explicitly target using the targeting tools 3.4 Isolating the role of delivery optimization
offered by Facebook about the user’s political leaning. Interestingly, Ad delivery is a complex process where multiple aspects can influ-
the effect does not persist when we use the FEC donor records ence the makeup of the audience that ultimately sees the ad. Here

18
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

CA𝐴 CA𝐵 CA𝐶 CA𝐷

DMA(s) [25]
Dem Rep Dem Rep Dem Rep Dem Rep
Greensboro, Charlotte 70,000 0 0 70,000 70,000 0 0 70,000
Wilmington, Raleigh-Durham,
0 63,137 70,000 0 0 54,000 64,166 0
Greenville-(New Bern and Spartanburg)
Table 1: Number of uploaded records for Custom Audiences created using publicly available voter records. We divide the DMAs
in the state into two sets, and create two audiences, each with voters registered with one party per DMA set (CA𝐴 and CA𝐵 ).
We repeat this process with separate voter records (creating CA𝐶 and CA𝐷 ). The number of uploaded records does not match,
as we uploaded records so as to achieve a match on the Estimated Daily Reach.

Sanders Trump Trump Sanders C L

Voter records Issues, Broad, 6

Issues, Narrow, 1
Donor records
Issues, Narrow, 7
0.2 0.3 0.4 0.5 0.6 0.7 0.8 Merch, Broad, 2
Estimated fraction of Democrats in the audience Merch, Broad, 3
Merch, Broad, 6
Figure 4: The estimated fraction of Democrats who were Merch, Narrow, 1
shown our ads, targeting both registered voters in North Car- Merch, Narrow, 4
olina and political donors. In the case of voter records, the Merch, Narrow, 8
ad delivery to Democrats ranges from 69% for Sanders’ ad to 0 0.5 1 10000 20000
only 39% for the Trump’s ad. In the case of donor records, we Estimated fraction Estimated
of Democrats in the audience audience size
do not see statistically significant differences in ad delivery.

Figure 5: We ran merchandise and issue ads with two levels

of targeting specificity (Broad: users with “Likely engage-
we discuss how we designed the experiments and analyzed the data ment with US political content (Conservative)” or “... (Lib-
in a way that limits the influence of factors other than the delivery eral)"; Narrow: additional detailed targeting for inferred in-
optimization itself. terest in Donald Trump or Bernie Sanders), and targeting
different regions (1: Celina, OH; 2: Dutchess, NY; 3: Lorain,
3.4.1 Role of competition. We ran each pair of campaigns targeting
OH; 4: Macclenny, FL; 5: McCormick, SC; 6: Richlands, VA;
a particular audience representing two different political campaigns
7: Saginaw, MI; 8: Slinger, WI). In all cases, Sanders’ ads de-
at the same time and with the same budget. Such a setup is designed
liver to a larger fraction of Democrats than Trump ads even
to ensure that both campaigns have the same users available for
though they are targeting the same audiences at the same
delivery (i.e., if run at different times, the skews could be attrib-
time using the same budgets. The effect is more pronounced
uted to different Facebook use patterns by liberals or conservatives)
for smaller audiences (compare, for example, Merch, Broad,
and both are experiencing the same competition from other ad-
3 and Merch, Broad, 6).
vertisers (i.e., that it would not be the case that one campaign is
under-performing because it happened to run at the same time that
another large and wealthy advertiser was targeting those users,
whereas another campaign avoided such a collision). Importantly, future delivery and pricing: reactions (e.g, ‘like’, ‘love’) , comment-
by selecting “reach” as the campaign optimization goal instead of ing, and sharing. Facebook’s advertising interface reports all such
“traffic”, we ensure that Facebook does not have the direct monetary engagements. Additionally, Facebook might be collecting and us-
incentive to show our ads to users who are more likely to click ing telemetric information; for example, how long each user spent
on them, especially in preference over other ads who might offer looking at the ad. This telemetric information is not available to
such incentive. Thus, running campaigns simultaneously is an ef- the advertisers (and thus, to us), but might still play a role in ad
fective strategy to isolate the effects of delivery optimization from delivery optimization algorithms.
other extraneous factors. Furthermore, to verify that the skews are Some of our ads received reactions, comments, and re-shares
not merely the effect of our ads competing with each other, we from users. We note three important, related observations, that
also re-ran a subset of campaigns separately. The qualitative and emphasize that our findings about skew in delivery and differential
quantitative skew effects for those campaigns were similar. pricing are not merely a function of the ad delivery algorithm’s use
of user engagement. First, we observe consistent skew and price
3.4.2 User engagement with our ads. There are a number of ways differences in ads that look identical to users, yet trick Facebook into
users can engage with ads, each of which potentially influences classifying them as partisan (Figure 2). Users do not react differently

19
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

to ads that appear identical, and, therefore, the entire observed public accountability or scrutiny, and raise open questions in the
difference can be attributed to Facebook’s relevance optimization domains of fairness and accountability in advertising.
(and some random effects). Second, we observe consistent skew in
delivery of ads that had virtually no engagement since they were 4.1 Implications
run on small budgets and only for a few hours (Figure 5). Third, First, Facebook limits political advertisers’ ability to reach audiences
we do find a negative correlation between the fraction of positive that do not share those advertisers’ political views in ways that
reactions (‘like’ and ‘love’) among all reactions and the price in are significantly different from traditional broadcast media. The
the longitudinal ads with 𝜌 = −0.91, 𝑝 𝑣𝑎𝑙 = 0.01. Taken together, existence and extent of this skew may not be apparent to advertisers
our work shows that although the skew in delivery and differential and varies based on their ad’s message as well as the destination link
pricing can be further amplified by users’ reactions, their primary used by the campaign. Furthermore, the strength of delivery skews
source is Facebook’s ad delivery optimization’s use of classification vary for campaigns of different political leanings and targeting
of the ad and its landing page content. different populations, making digital advertising inequitable for
political campaigns with identical budgets.
3.5 Limitations Second, recent moves to restrict political advertisers’ targeting
We note that we can only report on delivery skew that we observed options [7, 15, 16]—although valuable from a user privacy per-
for our own ads; we cannot draw any conclusions about how po- spective [12, 20, 29]—might be undermined by the ad delivery al-
litical ads in general (or all ads run by a particular campaign) are gorithms, and even give companies like Facebook more control
delivered. Nonetheless, the fact that we observe strong and statis- over selecting which users see which political messages. This selec-
tically significant effects in our small set of ads suggests that the tion may be occurring without the users’ or political advertisers’
potential negative outcomes for individuals, political campaigns, knowledge or control. Moreover, it is likely aligned with Facebook’s
and society in the context of ad delivery optimization of political business interests, but not necessarily with societal goals.
advertising are not mere hypotheticals and warrant further scrutiny. Third, today, researchers, regulators, and campaigns lack access
to algorithms and data required for a more thorough study of ad de-
Choice of partisan ads. In this work we chose to advertise two livery implications. In fact, Facebook has actively sought to thwart
candidates of two opposing American parties in order to clearly a recent initiative, NYU Ad Observer,8 whose goal was to collect
present the optimization of delivery to an audience also divided in such data [21]. Much has already been said about the inadequacy of
a binary way (liberal/conservative). However, even within the two current ad transparency tools provided by ad platforms for study-
parties there can be differing views, as exemplified by the process ing ad targeting [24, 31]. Our work draws attention to the need to
of primary elections, i.e., choosing one candidate among many to further expand these efforts to enable scrutiny of ad delivery. It is an
represent the party in general elections. Our current results do not interesting open question as to what algorithmic and data sharing
allow us to make strong statements about the potential differential advances are needed to enable such auditing while preserving user
pricing among different candidates of the same party. We leave this privacy and ad platform’s and advertisers’ competitive interests.
investigation, as well as auditing the differential pricing in systems
with more than two prominent political parties, to future work. 4.2 Policy analysis
Role of advertiser’s identity. We have repeated a subset of Today, U.S. law cannot do much, if anything, to directly change how
our experiments using another advertising account registered as platforms deliver political ads. For now, it is likely that the primary
an advertiser in the area of “Social Issues, Elections, and Politics” regulator of online political ads will not be the government, but
and linked to a Facebook page unrelated to the first. Our results rather ad platforms themselves.
were quantitatively and qualitatively similar. This suggests that The U.S. Congress has addressed conceptually similar “ad deliv-
the effects we observed were not tied to our particular advertising ery issues” in the past, albeit in a different domain. For example, the
account. Nevertheless, we do not make any statements about the Federal Communications Commission (FCC) enforces the so-called
extent to which the observed effects hold when run by real politi- Equal-Time Rule [1], which originated in 1927 in response to wor-
cal campaigns with a more established history and larger overall ries that broadcast licensees could unduly influence the outcome of
spending than ours. elections. The rule requires that licensees make air time available
to all candidates for the same office on equivalent terms. However,
Audience sizes. We aimed to match our constructed liberal and the rule only applies to broadcast licensees, and has only narrowly
conservative audiences in size as closely as possible, but the matches survived constitutional scrutiny in part because it implicates gov-
are inevitably imprecise as Facebook only provides estimates of daily ernment interests in managing limited broadcast spectrum [6].
reach7 rather than audience sizes. Regardless, we always ran both Prevailing interpretations of the First Amendment are likely to
liberal and conservative ads to the same audiences at the same time, block efforts to extend the logic of the Equal-Time Rule to digital ad-
so any imbalance in the audience size would affect both ads equally. vertising platforms, which are not regulated like broadcast licensees.
As an initial matter, the First Amendment strongly protects political
4 DISCUSSION speech, and generally tolerates only narrowly-tailored government
Our findings suggest that Facebook is wielding significant power regulations [34]. Moreover, the Supreme Court recently declared
over political discourse through its ad delivery algorithms without that “the creation and dissemination of information” constitutes
7 https://www.facebook.com/business/help/1691983057707189?helpref=faq_content 8 https://adobserver.org/

20
Session 1: Society WSDM ’21, March 8–12, 2021, Virtual Event, Israel

speech under the First Amendment [28]. This reasoning, which To Biased Outcomes. In ACM Conference on Computer Supported Cooperative
might expand the “commercial free speech” rights of companies, Work (2019).
[5] Angwin, J., and Parris Jr, T. Facebook Lets Advertisers Exclude Users By Race,
creates some uncertainty about the government’s ability to restrict 2016. https://www.propublica.org/article/facebook-lets-advertisers-exclude-
corporations’ use of data in digital advertising. users-by-race/.
[6] Cbs, Inc. V. Fcc. https://supreme.justia.com/cases/federal/us/453/367/, (Accessed
Looking ahead, it is clear that government regulation of digital on 2021-01-20).
political advertising is on firmest legal footing when it requires dis- [7] Conger, K. Twitter Will Ban All Political Ads, C.E.O. Jack Dorsey Says,
closure about who is speaking to whom, when, and about what [34]. 2019. https://www.nytimes.com/2019/10/30/technology/twitter-political-ads-
ban.html.
Accordingly, Congress and the FEC can consider transparency re- [8] Creating customer lists with mobile advertiser ids. https://developers.facebook.
quirements that will enable detailed auditing of ad targeting and com/docs/app-ads/targeting/mobile-advertiser-ids/, (Accessed on 2021-01-20).
the delivery optimization as applied to political ads. [9] Eckles, D., Gordon, B. R., and Johnson, G. A. Field Studies Of Psychologically
Targeted Ads Face Threats To Internal Validity. PNAS: Proceedings of the National
Academy of Sciences of the United States of America 115, 23 (2018), 5254–5255.
4.3 Mitigations [10] Facebook: About Facebook Pixel. https://www.facebook.com/business/help/
742478679120153, (Accessed on 2021-01-20).
The public, policy makers, researchers, and the campaign managers [11] Facebook Ad Library. https://www.facebook.com/ads/library/, (Accessed on
2021-01-20).
need more information about the operation of ad delivery algo- [12] Faizullabhoy, I., and Korolova, A. Facebook’s Advertising Platform: New
rithms and their real-world effects. Ad platforms could increase Attack Vectors And The Need For Interventions. https://arxiv.org/abs/1803.10099,
transparency around political ads (including key metrics such as Workshop on Technology and Consumer Protection (ConPro 2018).
[13] Fowler, E. F., Franz, M. M., Martin, G. J., Peskowitz, Z., and Ridout, T. N.
targeting criteria, detailed ad metadata, ad budgets, and campaign Political advertising online and offline. American Political Science Review (2020).
objectives) to enable further study of the effects of ad targeting [14] Gelauff, L., Goel, A., Munagala, K., and Yandamuri, S. Advertising for
and delivery. And they could provide access to and insight into demographically fair outcomes, 2020. https://arxiv.org/abs/2006.03983.
[15] Glazer, E. Facebook Weighs Steps To Curb Narrowly Targeted Political Ads,
the ad delivery algorithms themselves (including those involved in November 2019. https://www.wsj.com/articles/facebook-discussing-potential-
running the auction, relevance measurement and estimation, and changes-to-political-ad-policy-11574352887.
[16] Google: An Update On Our Political Ads Policy. https://www.blog.google/
bid and budget allocation on advertisers’ behalf), allowing third technology/ads/update-our-political-ads-policy, (Accessed on 2021-01-20).
parties greater ability to study and audit their performance and ef- [17] Green, J., and Issenberg, S. Inside The Trump Bunker, With Days To Go,
fect on political discourse. Ad platforms could also disable delivery 2016. https://www.bloomberg.com/news/articles/2016-10-27/inside-the-trump-
bunker-with-12-days-to-go.
optimization for political content, or at least allow advertisers to do [18] Imana, B., Korolova, A., and Heidemann, J. Auditing for bias in algorithms
so. They could also introduce more nuanced user-facing controls delivering job ads. In Proceedings of The Web Conference (2021).
for political content delivery. Beyond these mitigations, our work [19] Kayser-Bril, N. Automated discrimination: Facebook uses gross stereotypes to
optimize ad delivery. AlgorithmWatch (2020).
highlights the need for advances that could help set the goals of ac- [20] Korolova, A. Privacy Violations Using Microtargeted Ads: A Case Study. Journal
countability, fairness, and interpretability in advertising delivery on of Privacy and Confidentiality 3, 1 (2011), 27–49.
[21] Lyons, K. Facebook wants the NYU Ad Observer to quit collecting data about its
firm scientific ground. Finally, we call on ad platforms to acknowl- ad targeting. The Verge (2020).
edge the central role they play in the delivery of political ads, and [22] Matz, S. C., Kosinski, M., Nave, G., and Stillwell, D. J. Psychological Targeting
to collaborate with other key stakeholders—including researchers, As An Effective Approach To Digital Mass Persuasion. PNAS: Proceedings of
the National Academy of Sciences of the United States of America 114, 48 (2017),
political campaigns, journalists, law, and policy scholars—to address 12714–12719.
that role when it is not aligned with public interests. [23] Merrill, J. Facebook Charged Biden a Higher Price Than Trump for Campaign
Ads, October 2020. https://themarkup.org/election-2020/2020/10/29/facebook-
political-ad-targeting-algorithm-prices-trump-biden, (Accessed on 2021-01-20).
ACKNOWLEDGEMENTS [24] Mozilla. Facebook’s Ad Archive Api Is Inadequate, April 2019. https://blog.
mozilla.org/blog/2019/04/29/facebooks-ad-archive-api-is-inadequate/.
We would like to thank Dean Eckles and David Lazer for their [25] Nielsen dma® regions. https://www.nielsen.com/intl-campaigns/us/dma-maps.
invaluable insights. We are also grateful to the participants and html, (Accessed on 2021-01-20).
[26] Reach Objective. https://www.facebook.com/business/help/218841515201583,
organizers of the REAL ML workshop for their encouragement and (Accessed on 2021-01-20).
constructive feedback. This work was done, in part, while Alek- [27] Ribeiro, F. N., Saha, K., Babaei, M., Henriqe, L., Messias, J., Oana Goga, F. B.,
sandra Korolova was visiting the Simons Institute for the Theory Gummadi, K. P., and Redmiles, E. M. On Microtargeting Socially Divisive Ads:
A Case Study Of Russia-linked Ad Campaigns On Facebook. In Conference on
of Computing, where she benefited from feedback of participants Fairness, Accountability, and Transparency (Atlanta, Georgia, USA, January 2019),
in the Privacy and Fairness programs, and particularly, from the ACM, pp. 140–149.
[28] Sorrell V. Ims Health Inc. https://supreme.justia.com/cases/federal/us/564/552/,
suggestions of Amos Beimel and Kobbi Nissim. This work was (Accessed on 2021-01-20).
funded in part by a grant from the Data Transparency Lab, NSF [29] Speicher, T., Ali, M., Venkatadri, G., Ribeiro, F. N., Arvanitakis, G., Ben-
grants CNS-1616234, CNS-1916020, CNS-1916153, CNS-1956435, evenuto, F., Gummadi, K. P., Loiseau, P., and Mislove, A. On The Potential
For Discrimination In Online Targeted Advertising. In Conference on Fairness,
CNS-1943584, and Mozilla Research Grant 2019H1. Accountability, and Transparency (New York, New York, USA, February 2018).
[30] United States V. Internet Research Agency. https://www.justice.gov/file/1035477/
download, (Accessed on 2021-01-20).
REFERENCES [31] Upturn. Leveling The Platform: Real Transparency For Paid Messages On
[1] 47 Usc §315 – Candidates For Public Office. https://www.law.cornell.edu/uscode/ Facebook, 2018. https://www.upturn.org/reports/2018/facebook-ads/.
text/47/315, (Accessed on 2021-01-20). [32] Venkatadri, G., Sapiezyński, P., Redmiles, E. M., Mislove, A., Goga, O.,
[2] Ads About Social Issues, Elections Or Politics. https://www.facebook.com/ Mazurek, M., and Gummadi, K. P. Auditing Offline Data Brokers Via Face-
business/help/208949576550051, (Accessed on 2021-01-20). book’s Advertising Platform. In International World Wide Web Conference (2019).
[3] Agresti, A., and Coull, B. A. Approximate Is Better Than “exact” For Interval [33] Weintraub, E. L. Don’t Abolish Political Ads On Social Media. Stop Microtar-
Estimation Of Binomial Proportions. The American Statistician 52, 2 (1998), geting., November 2019. https://www.washingtonpost.com/opinions/2019/11/01/
119–126. dont-abolish-political-ads-social-media-stop-microtargeting/.
[4] Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., and Rieke, A. [34] Wood, A. K., and Ravel, A. M. Fool Me Once: Regulating Fake News And Other
Discrimination Through Optimization: How Facebook’s Ad Delivery Can Lead Online Advertising. Southern California Law Review 91 (2017), 1223.

21