Who gets to choose when decisions are made with AI? That was the question of my lightning talk at AI4HS 2024, the WASP-HS conference Creating Shared AI Futures in Göteborg. The short answer is that choices are not made in one place. They are distributed across code, data, people, time and place. And that makes questions of power and agency harder—and more interesting—than the debate about fair algorithms suggests.
Beyond the technical fix
Much of the debate about AI and fairness is framed as a technical problem. Make the algorithm transparent. Make it explainable. Fix the math. This is a convenient solution, for ethicists and system builders alike. But it shrinks the problem to the computational realm.
And there are limits to what the math can do. As the mathematician David Sumpter shows in Outnumbered, models struggle with heterogeneous populations. Some problems cannot be fixed inside the model.
Choices built into infrastructure
AI systems are not built from scratch. They are layered on top of other systems, each with its own assumptions and constraints. Kristian Bondo Hansen and Nanna Thylstrup call this stack bricolage.
The “left-pad” incident of 2016 shows what this means. After a dispute with the messaging company Kik over a package name, the developer Azer Koçulu withdrew all his code from the npm software repository. Among it was “left-pad,” a package of eleven lines. Thousands of projects depended on it. When it disappeared, major platforms and applications broke around the world.
A single developer’s local choice rippled through the global software supply chain. That is our computational reality today.
Randall Munroe, xkcd 2347: Dependency, CC BY-NC 2.5.
Choices built into data
Data does the same thing across time. Katrin Amelang and Susanne Bauer have followed how epidemiological risk scores travel. The Framingham Heart Study began in 1948, in a predominantly white, middle-class town in Massachusetts. Today its data is built into German primary care software, where it calculates and displays cardiovascular risk scores for patients.
Seventy years of changes in populations, lifestyles and living conditions—and the data is treated as if it were just as relevant today. Choices made in 1948 shape decisions in a doctor’s office now. That is our data reality today.
So where is agency?
This is a wicked problem. When choices are distributed across global infrastructures and decades of data, where do we even look? What is the site of a decision?
For STS, this raises two questions. How do we study these distributed assemblages? And how might we design them so that the right people have a voice at the moments that matter?
I think the challenge goes beyond making algorithms “fair” or “ethical.” It is about how computational infrastructures reshape the possibilities for human choice. That means moving beyond neat technical solutions, and getting into the messy work of making machine learning work in practice. I have tried to do some of that in Enacting the pandemic, and with Lotta Björklund Larsen in How should we theorize algorithms?
References
Amelang, Katrin, and Susanne Bauer. 2019. “Following the Algorithm: How Epidemiological Risk-Scores Do Accountability.” Social Studies of Science 49 (4): 476–502. https://doi.org/10.1177/0306312719862049
Hansen, Kristian Bondo, and Nanna Thylstrup. 2024. “Stack Bricolage and Infrastructural Impermanence in Financial Machine-Learning Modelling.” Journal of Cultural Economy 17 (1): 20–38. https://doi.org/10.1080/17530350.2023.2229347
Sumpter, David. 2018. Outnumbered: From Facebook and Google to Fake News and Filter-Bubbles – The Algorithms That Control Our Lives. London: Bloomsbury Sigma.