Datajusticelab Critical Data Literacy Guidebook
Datajusticelab Critical Data Literacy Guidebook
June 2020
1
Critical data literacy tools for advancing data justice: A guidebook
About this guidebook
This guidebook is intended to provide anyone interested in critical data literacy with an up-
to-date overview of a range of different types of data literacy tools available, keeping a
diversity of audiences and skill levels in mind. The literacy tools in this guidebook are
resources that are freely available for anyone to use and which educate citizens about
datafication and its social consequences, by explaining relevant topics in an engaging way
and by facilitating active participation in datafied society.
For each tool page there is a brief explanation of how the tool works and how to use it,
followed by a suggestion of which audiences (non-experts, civil society, communities,
teachers etc.) and contexts the tool is most suitable for based on what the tool creators
have indicated. For instance some of the tools can be used widely, whereas others have
more specific applications. Further, the “What you’ll need” section indicates if any prior skills
are necessary to use the tool, whether additional resources such as pens, paper and print-
outs are required and, where applicable, how much time is needed to use the tool. Finally,
every tool page has a “Literacy outcome” section that highlights what citizens will gain from
using each tool, for example greater critical understanding or privacy tips that can be
implemented on their device(s).
Tool selection rationale
All fourteen tools in this guide have been selected on the basis that they contribute to
critical data literacy in some way, broadly defined as the ability to critically engage with
datafication by reflecting on the societal implications of data processing and implementing
this understanding in practice.1 Other tools were selected, however, because they address
current gaps or shortfalls that we have observed in approaches to data literacy such as
attention to how collective dynamics can be encouraged, mechanisms for civic participation,
understandings of public sector uses of data and algorithms, and practical applications of
critical thinking.
Typology of tools
There are six categories of literacy tools in this guidebook, defined in terms of how they
educate or engage citizens in relation to data issues: through workshops, interactive
learning, investigations, using data for participation, quick practical guides and longer in-
depth guides. These definitions are overlapping, however, as some of the tools could be
placed in more than one category.
1 The guidebook draws from and updates previous research by Sander. For a discussion of ‘critical big data
literacy’ see Sander, I. (2020). What is critical big data literacy and how can it be implemented?. Internet Policy
Review, 9(2). DOI: 10.14763/2020.2.1479. For a comprehensive mapping and commentary of data literacy tools
see research by Sander, including some of the tools that were defined as ‘critical big data literacy tools’ in her
‘Critically Commented Guide to Data Literacy Tools’ (2019) available at:
https://zenodo.org/record/3241422#.XuJpw1VKjIU.
2
1. Workshop resources
These can be used to facilitate workshops and are best for groups and communities because
they encourage collective thinking about data. The Digital Defense Playbook (page 4) helps
affected communities to think about collective strategies of resistance to and protection
from the surveillance state. The Unbias Fairness Toolkit and Youth Jury Resource Pack
(page 5) can be used to facilitate workshops and jury-style discussions on the topic of
algorithmic bias and fairness. The Algorithmic Ecology (page 6) is an analytical tool and
framework for organising community resistance to algorithms.
2. Interactive learning tools
The three tools in this category educate citizens by visualising or simulating data driven
systems to create an interactive experience. Automating NYC (page 7) is an interactive
website that uses real-world examples of algorithms to visually explain how they work and
the real impact they had on New York City residents. Do Not Track (page 8) is a short
interactive documentary series that uses the viewer’s data to explain online tracking. Pre-
Crime: Predictive Policing Simulator (page 9) also uses real-world algorithms to simulate a
gamified predictive policing experience in which the viewer is both the citizen being scored
by “Agatha” the algorithm, and the data scientist trying to perfect the system to reduce
crime.
3. Investigation tools
Data Scores Investigation Tool (page 10) and Algorithm Tips (page 11) are designed to help
citizens, and especially researchers and journalists, to further investigate public sector uses
of data systems in the UK and US respectively. They are both accessible databases with easy-
to-use search functions.
4. Participation tools from algorithmic accountability projects
These two research projects invite citizens to participate in holding platform algorithms to
account. Algorithms Exposed (page 12) and Algorithm Watch (page 13) have both
developed browser plug-ins that anyone can download and which work by “donating” user
data from Amazon, YouTube, Facebook and Instagram to the projects. Algorithms Exposed is
particularly useful for researchers and data scientists who are interested in personalisation
algorithms and can use the project’s Github to examine results from the project.
5. Quick guides for implementing stronger privacy
These are a set of technical steps that citizens can easily and quickly implement to optimise
settings on their device(s) for stronger data privacy. I Have Something To Hide (page 14)
offers a “data protection toolkit” and Privacy International (page 15) have produced a guide
to minimising targeted advertising on nine different platforms. These tools are good for
providing individuals with more privacy and control.
6. In-depth guides to data protection and the data economy
These are comprehensive guides which require more reading and provide thorough
introductions to their given topic, but which also have a focus on the practical application of
the learning they offer. Your Data Your Rights (page 16) guides citizens through the
fundamental GDPR rights using text and video, and provides template letters to send to data
controllers to exercise these rights. Me and My Data Shadow (page 17) is no longer being
updated but still offers a comprehensive overview of what digital footprints are and how
they can be used to discriminate, while also offering a number of practical ways to control
data shadow “traces” and resources for educators like training session plans.
3
1. Workshop resources
A). Digital Defense Playbook: Community Power Tools for Reclaiming Data
A workbook of popular education activities focused on data, surveillance and community safety
that can be used to co-create tools and knowledge for data justice.
How it works
The Algorithmic Ecology framework is presented as a template worksheet that enables
activists, organisers and communities to visually map all of the actors involved in or affected
by the deployment of a particular algorithm. The framework divides an algorithm’s ecology
into four layers: ideological, institutional, operational and community, enabling organisers to
decentre the technology and think about where and how to target resistance.
Literacy outcome
√ This tool provides you with a critical understanding of the origins and societal
implications of tracking.
2 This resource was also the most popular among participants in Ina Sander’s study of citizen engagement with
data literacy tools. Although the series was made in 2015 it is still relevant for today’s big data economy.
8
C). Pre-crime calculator: A predictive policing simulator
An immersive role-playing experience that shows
the player how predictive policing works, and
which accompanied the release of the 2017
documentary “Pre-Crime” by Matthias Heeder
and Monika Hielscher.
Tool type: Simulated experience
Creators: Michaela Pnacekova (author)
Kloos & Co Medien (production) (2017)
URL: https://kloosundco.de/en/pre-crime-
calculator-2/
How it works
Visitors can play this simulation via their browser or an app and the user journey begins with
a “Prologue” in which an algorithm named “Agatha” asks the player questions to score them
based on inferences from their interrogation by Agatha and real personal data that they
provide*. These data are used to calculate the player’s “danger-potential” based on real
predictive policing algorithms. The second part of the simulation is the “gamified
experience” in which the player plays the role of data scientist trying to perfect the
algorithm in order to reduce crime in their real location.
Who is this tool good for?
√ Teachers, since this tool works well in
a classroom setting because it’s
interactive and the experience of
being scored by “Agatha” is very
engaging.
√ Facilitating group discussions among
friends, students, colleagues and
workshop participants about
predictive policing as well as the
ethical, political and social justice
implications of algorithmic decision-
making systems.
√ Public sector workers who work with
data scoring systems.
What you’ll need
√ The simulation works on a laptop, tablet or mobile but is not available for Android.
Literacy outcomes
√ Critical understanding of how the inferences of scoring systems can lead to biased
outcomes and entrench existing prejudices.
√ Awareness of the complex moral dilemmas that predictive policing highlights.
√ Critical reflexivity on the ways in which scoring systems are used to make decisions and
predict human behaviour.
*This data is collected for the purpose of using the simulated experience only. It is not stored anywhere or
9
shared with anyone.
3. Tools for investigating public sector data systems
A). Data Scores investigation tool
As part of the Data Justice Lab’s ‘Data Scores as Governance’ project this tool was created to
map and investigate the uses of data analytics and algorithms in public services in the UK.
Tool type: Investigation tool
Creator: Data Justice Lab (2018)
URL: https://data-scores.org/
How it works
Four different search functions allow visitors to explore the site’s database of more than six
thousand documents relating to UK public sector data systems. The Insights function enables
the visitor to explore data systems by department, location, organisation and software type
in a visually appealing way, while the entire database can be searched by key word or phrase
under the Documents Index. The Case Studies section offers further details on six data
systems that were researched by the Lab, and the Overviews section provides visitors with a
visual map of known predictive analytics systems in public services across the UK.
10
B). Algorithm Tips
The Data Scores tool was modelled on this similar online tool which enables anyone to
investigate American federal government uses of automated systems.
Tool type: Investigation tool
Creator: Nick Diakopoulos,
Northwestern University (2017)
URL: http://algorithmtips.org/
How it works
The website is primarily a
database of government
algorithms which is easily
searchable by any word that’s in
a metadata field, for example
“child welfare” or “2019”.
Information for each algorithm
is displayed in a simple format
and has a link to the
government agency that is using
it.
Under the resources tab there is
an extensive list of links to
journalistic investigations of
algorithms, methods for
auditing and critiquing
algorithms and tips for how to
submit a Freedom Of Information request for algorithms. There is also a blog that discusses
implications of some of the algorithms.
Who is this tool good for?
√ Journalists and researchers who want to find out more about current government uses
of algorithms. The site was created to be a starting-point resource for algorithmic
accountability projects in particular.
What you’ll need
√ This website is easy to use so
you won’t need any specific
skills.
Literacy outcomes
√ Users gain leads for
journalistic or academic
investigations.
√ Journalistic skills for
investigating and critiquing
algorithms (see resources tab).
11
5. Algorithmic accountability projects
A). Algorithms Exposed (ALEX)
A side project from the University of Amsterdam’s DATACTIVE project that is focused on
developing methods for auditing proprietary algorithms, particularly aiming to uncover the
personalisation algorithms at work on Amazon, YouTube and Facebook while also
encouraging data activism.
Tool type: Algorithmic accountability project
Creators: Claudio Agosti (lead developer);
DATACTIVE, University of Amsterdam (2019)
URL: https://algorithms.exposed/
How it works
Resources for the three platform audits are not
immediately obvious on the project website but
they can be found under the Software section of
the Output tab. Here there are links to the
software tools that anyone can download,
customise and use to examine tracking and
personalisation on Amazon, Facebook and
Youtube, including browser extensions for each
platform. There are step-by-step guides for
installing the extensions. Using cryptography,
these extensions send user data to ALEX’s server
and are fully GDPR compliant, but users have full Who is this tool good for?
control over which data they choose to volunteer √ Experts such as data scientists, analysts and
to the project. Receiving this data enables the researchers working on the topic of
project team to compare personalised social algorithmic accountability and personalisation
media experiences from different internet users. algorithms (especially the YouTube and
Amazon tools which are research-oriented).
What you’ll need √ Journalists and policy makers who can make
√ To participate you’ll need a YouTube, Amazon use of the project’s auditing methods.
or Facebook account. √ Curious citizens who can use the resources to
√ To be able to play around with the tools and gain more control over their Amazon, YouTube
compare results, data analysis and coding and Facebook experiences.
skills will be useful (Python and R). √ Keep in mind there’s a lot of information and
links on this site so it can be a bit confusing to
navigate.
Literacy outcomes
√ Critical reflexivity on your own information
diet and social media use.
√ Critical evaluation of personalisation
algorithms.
√ Learn how to use your data to participate in
civic projects.
12
B). Instagram algorithm data donation plug-in
Developed by Algorithm Watch, this is a similar participatory tool to Algorithms Exposed that
also uses data “donation” as a method to research Instagram’s algorithm, which has been
under-researched compared to Facebook and YouTube.
Tool type: Algorithmic accountability project
Creator: Algorithm Watch (2020)
URL: https://algorithmwatch.org/en/instagram-
algorithm/
How does it work?
By visiting the project page on Algorithm Watch’s website, users can participate by
downloading the plug-in for Firefox. Users will then be asked to follow three specific
Instagram accounts and continue to use the platform as normal. The plug-in collects
information about the pictures and videos that appear in users’ newsfeeds plus information
about some of the accounts that they follow. All the information is anonymised in a way that
makes it impossible for Algorithm Watch to re-identify data donors. What Algorithm Watch
wants to find out is whether Instagram favours some types of content over others.
Who is this tool good for?
√ Instagram users who want to contribute to algorithmic accountability.
√ Academics, researches or data scientists interested in the topic of personalisation
algorithms, as you can create an account just for the experiment and don’t need to be
active for it to work.
√ It’s not yet clear how AW will communicate the results they receive from people
donating their data so the literacy potential is not entirely clear.
What you’ll need
√ To participate you need Firefox browser and an Instagram account, but no prior skills or
knowledge.
√ Literacy outcome: Encourages participation and may prompt critical evaluation.
13
6. Quick privacy guides
A). I have something to hide
15
In-depth guides to the GDPR and data economy
A). Your Data Your Rights
A practical guide to the GDPR that aims to empower EU citizens to understand and
ultimately use their data protection rights.
Tool type: In-depth guide to data
protection
Creator: Digitale Gesellschaft e.V. (2018)
URL: https://yourdata-yourrights.eu/
How it works
This easy to navigate site explains
fundamental GDPR rights and principles
in four different ways: a game that
invites you to defend your data against
unfair attacks; accessible articles that
provide overviews of each right; short
animated videos that explore data
protection in every day life; and different
sample letters that can be used as a
template to exercise each GDPR right.
There is also a dictionary which goes into
more detail on specific aspects of some
of the rights and rules.