Tyranny of the Algorithm? Predictive Analytics & Human Rights
From predictive policing to the social media tracking of potential terrorists, governments worldwide rely increasingly on data-driven risk assessments. On March 21-22, the Bernstein Institute hosted its second annual conference to address the human rights implications of this trend. Tyranny of the Algorithm? Predictive Analytics & Human Rights, featured over 30 speakers, including Samuel Rascoff, NYU professor and former director of intelligence analysis at the NYPD, and Latanya Sweeney, Harvard professor and former Chief Technologist at the Federal Trade Commission. The conference was attended by over 200 academics and practitioners. Consistent with the inter-disciplinary mission of the Institute, speakers and attendees came from a range of backgrounds, with participants including legal academics, computer scientists, business leaders, anthropologists and law enforcement officials.
Some speakers suggested that data-driven risk assessments may be a promising way to mitigate racial, gender, or religious bias in policing and counter-terrorism operations. Jeff Brantingham, UCLA professor of anthropology and co-founder of the predictive policing technology, PredPol, argued that algorithms can more accurately and objectively predict certain crime “hot spots” than human analysts. But the technology comes with risks. Patrick Ball, co-founder of Human Rights Data Analysis Group, explained how machine learning will get the wrong answers when unrepresentative data is fed into predictive algorithms. Jason Q. Ng, a research fellow at The Citizen Lab and data analyst at Tumblr raised the potential human rights concerns about the use of seemingly benign predictive technology by the Chinese government.
Speakers also addressed ways to improve human rights accountability in the field of predictive analytics. One approach, discussed by Frank Romo, an analyst with Million Hoodies for Justice, is for communities to use the tools of predictive analytics to monitor government activity. Just as predictive technology can help police anticipate sites of future crime, so too can the technology be used to predict sites of future police violence. And Sorelle Friedler, a computer science professor at Haverford College and fellow at Data & Society, spoke about her work on building fair algorithms.
The meeting opened with two framing discussions. Presenters focused on core concepts, controversies, and dynamics to ensure a common vocabulary for participants from a range of backgrounds.
Framing Discussion #1: Human Rights Challenges in Predictive Analytics
Moderator:
- Margaret Satterthwaite ’99 – Professor, NYU School of Law; Faculty Director, Bernstein Institute
Speakers:
- Jeff Brantingham – Professor, UCLA, Department of Anthropology, Chief of Research and Development, Predpol
- Rachel Levinson-Waldman – Senior Counsel, Liberty and National Security Program Brennan Center for Justice at NYU Law School
- Jennifer Lynch – Senior Staff Attorney, Electronic Frontier Foundation
- Latanya Sweeney – Professor, Harvard University
This opening panel defined key terms and framed the conversation about data-driven risk assessment as one that is global in scope. The Panel emphasized the value of a human rights lens for uncovering key dynamics by introducing real-world examples of the human rights impact of data-driven risk assessment in the three contexts that the conference focused on: crime prevention and social control, pre-trial detention, and policing of violent extremism and terrorism. It considered the capacity of human rights practitioners to access, evaluate, and challenge data-driven risk assessments by government actors and their private sector partners involved in predicting and preventing prohibited activity.
Framing Discussion #2: Human Rights and the Normative Dimensions of Predictive Analytics
Moderator:
- Sharon K. Hom ’80 - Executive Director, Human Rights in China; China and International Human Rights Law Research Program, Robert L. Bernstein Institute
Speakers:
- Mark Latonero – Fellow, Data & Society; Senior Research Fellow, Annenberg Center on Communication Leadership & Policy
- Sally Merry – Silver Professor of Anthropology, NYU; Faculty Co-Director, Center for Human Rights and Global Justice, NYU School of Law
- Helen Nissenbaum – Professor, NYU, Department of Media, Culture, and Communication & Computer Science, Director, Information Law Institute
- Mimi Onuoha – Artist, Fellow, Data & Society
This panel reflected on the social, political, economic and cultural context in which data-driven risk assessments are designed and implemented. This discussion made explicit the often implicit normative choices behind data-driven risk assessments and emphasize the human rights implications flowing from those choices.
NYU Alumni Panel: Working with Predictive Analytics
Moderator:
- Margaret Satterthwaite ‘99 – Professor, NYU School of Law; Faculty Director, Bernstein Institute
Speakers:
- Baher Azmy ’96 – Legal Director, Center for Constitutional Rights, Professor, Seton Hall University, School of Law
-
Alex Chohlas-Wood ’14 – Director of Analytics, New York Police Department
- Ahmed Ghappour ’07 – Visiting Assistant Professor and Director, Liberty, Security and Technology Clinic, UC Hastings Law
- Elizabeth Joh ’00 – Professor, UC Davis School of Law
March 22, 2016
The morning consisted of two conversations addressing cross-cutting challenges that arise in data-driven risk assessment. The afternoon consisted of three case-study panels to facilitate a deeper-dive into how human rights accountability could be achieved in data-driven risk assessments for specific purposes.
Cross-Cutting Panel #1: Data Collection and Ownership: The Role of the Private Sector
Moderator:
- Michael Posner – Professor, NYU Stern School of Business, Co-Director, Stern Center for Business and Human Rights
Speakers:
- Sarah Labowitz – Co-Director, NYU Stern Center for Business and Human Rights
- Andrew McLaughlin – Partner, Betaworks
Much of the data used by governments in their risk assessment activity is collected, stored, analyzed, and sometimes owned by corporations. This panel considered how human rights principles can guide decision-making by corporations who partner with government agencies in predictive analysis, and what role corporations can and should play in upholding the rights of those who are subject to these analyses. In exploring these questions, panelists worked through real-world examples that have occurred with respect to crime prevention and social control, pre-trial detention, and policing of extremism and terrorist threats.
Cross-Cutting Panel #2: Data Hygiene and Algorithmic Oversight
Moderator:
- Sorelle Friedler – Assistant Professor of Computer Science, Haverford College; Fellow, Data & Society
Speakers:
- Patrick Ball – Co-founder, Director of Research, Human Rights Data Analysis Group
- Frank Pasquale – Professor, University of Maryland School of Law
-
Kristian Lum – Lead Statistician, Human Rights Data Analysis Group
Data-driven risk assessments raise difficult issues of oversight, since they rely on data sets and algorithms that may be hidden, opaque, or misunderstood by the general public. Many of the actors who usually play a role in the detection of human rights violations by governments do not have the technical expertise to understand, let alone challenge, the quality of the data and fairness of the algorithms behind data-driven risk assessments. This panel focused on how this gap in the human rights accountability landscape can be overcome. In exploring these questions, panelists worked through real-world examples that have occurred with respect to crime prevention and social control, pre-trial detention, and policing of terrorist threats.
From the Actuarial to the Algorithmic: Virtual transparency in our expository society
Presenter:
- Bernard Harcourt - Isidor and Seville Sulzbacher Professor of Law; Director, Columbia Center for Contemporary Critical Thought, Columbia Law School
Case-Study #1: Predictive Analytics for Crime Prevention and Social Control
Moderator:
- Katherine Strandburg – Alfred B. Engelberg Professor of Law, NYU School of Law
Speakers:
- Sarah Brayne – Post-doctoral Researcher, Microsoft Research
-
Ritesh Kotak – Strategic Advisor to law enforcement
- Jason Q. Ng – Research Fellow, Citizen Lab; Data Analyst, Tumblr; Lecturer, Columbia University
- Frank Romo – Data Analyst, Million Hoodies for Justice
Police in different parts of the world are using data-driven risk assessment tools for crime prevention and social control. In open societies, it is hoped these tools will predict and help prevent violence before it happens. Under repressive regimes, the same tools are being adopted to monitor and control risks of social unrest. Serious concerns have been raised about the disparate impact of these tools on marginalized communities. This panel looked at the human rights challenges of data-driven risk assessment for crime prevention and social control, and explored whether the participation of policed communities could help mitigate these challenges in some contexts.
Case-Study #2: Predictive Analytics for Pre-Trial Detention
Moderator:
- Ira Belkin ‘82 - Executive Director, U.S.-Asia Law Institute, NYU School of Law
Speakers:
- Solon Barocas – Post-doctoral Researcher, Center for Technology Policy, Princeton University
- Fu Hualing – Professor, University of Hong Kong
- Anne Milgram ‘96 – Distinguished Scholar in Residence, NYU School of Law
- Dan Wei – Professor of the Institute Procuratorial Theory of the Supreme People’s Procuratorate, China
The criminal justice system is using data-driven risk assessments to make determinations of dangerousness and flight risk to guide decisions about whether to place individuals in pre-trial detention. This panel looked at the human rights challenges of data-driven risk assessments by legal actors, and considers the ability of human rights advocates and defense lawyers to evaluate and challenge these assessments.(No video available).
Case-Study #3: Predictive Analytics for Policing “Violent Extremism”
Moderator:
- Samuel Rascoff – Professor, NYU Law School
Speakers:
- Didier Bigo – Professor, King’s College, London; Research Professor (MCU) Sciences Po Paris; Directeur of the Centre d’études sur les Conflits, la Liberté, la Sécurité (CCLS)
-
Faiza Patel – Co-Director, Liberty and National Security Program, Brennan Center for Justice, NYU School of Law
- David Suter – Ph.D., Zurich University
Police departments, tasked with predicting and preventing the emergence of terrorist violence, are relying on data-driven risk assessment tools designed to help counter violent extremism, drawing in data obtained through widespread surveillance systems worldwide. This panel looked at the human rights challenges of data use for the prediction of this kind of violence. And it interrogated the ability of human rights practitioners to evaluate and challenge the use of data-driven risk assessments in this context.(No video available).