This is my talk at The Culture Code*, organized by Kıvanç Tatar at Chalmers in Göteborg on 13 February 2026. The images are slides from the talk; each cites its source.*
What is the relationship between the world, data and representation? That question has followed me through two projects on synthetic data: Social Complexity and Fairness in Synthetic Medical Data, with Ericka Johnson, Gabriel Eilertsen, Saghi Hajisharif and Tahereh Dehdarirad, and now the WASP-HS research group Synthetic Data: Facts, Representations, and Transparency, with Ylva Söderfeldt, Ericka Johnson and our team.
Wait… what?
In a recent paper, Dietmar Offenhuber writes that with synthetic data “we are confronted with data that does not involve any representational relationships.”
Wait… what?
Researchers have shown that diffusion models can reproduce images from their training data—almost pixel for pixel. Put the original next to the generated image, and it is hard to say that there is no representational relationship. So what is the relationship between the world, data and representation? And what does it mean to represent something?
Original training images and images generated by a diffusion model. From Carlini et al. 2023, “Extracting Training Data from Diffusion Models.”
Philosophers talk about truthlikeness, or verisimilitude: how close to the truth something is. (Thanks to Tanya Osborne at Chalmers for this line of thinking.) But STS has a different starting point.
Data are made in practice
STS has long shown that data are fundamentally made in practice. Or, as Lisa Gitelman put it, “raw data” is an oxymoron. A few classics show how:
- Seeing. John Law and Michael Lynch showed how bird watchers learn to see the world and tell objects apart with lists and field guides. Which representations of the world are best to see with?
- Data and evidence. Karin Knorr Cetina and Klaus Amann showed how scientists fix visual traces as evidence. In the lab, autoradiographs are cut, pieced together, labeled and pointed at before they become evidence.
- Sublata. Bruno Latour followed soil scientists in the Amazon, from world to samples, traces, measurements, inscriptions, figures and arguments. Representation is made through chains of transformations.
- Capta. Johanna Drucker reminds us that data contain epistemologies: data are given, capta are taken.
- Crafting. Janet Vertesi showed how images are crafted to see specific things.
So instead of asking whether synthetic data represents the world, we can ask: how is synthetic data made to represent the world in practice? We can sketch its chain of transformations—and ask where the human is in the loop.
Without pretraining, there is only noise
Try it yourself. Prompt GPT-2 without pretraining with “The meaning of life is:” and you get a random string of tokens: “sinners blows1500 ancestry rarelections WinnerLEY wreckage Salary…” The pretrained GPT-2 answers with a sentence: “to make a distinction between things we want to be, and people we cannot make real…” The same goes for image models. Stable Diffusion without pretraining gives you noise. Pretrained, it gives you a futuristic city with neon lights.
The world is in the data. Pretraining is a step in the chain of transformations—not a break in it.
Model cultures and architectural assumptions
The chain is also shaped by the models themselves. As one computer science professor told me in an interview: “Certain groups specialize in particular architectures, become proficient in them, and then continue working with those.” And: “Different architectures are suited to different types of data… convolutional neural networks… [are] developed on the basis of certain a priori assumptions that images have a specific structure.”
Model cultures and architectures carry assumptions about what the world looks like.
Ground truthing
And then there is ground truth. Florian Jaton has shown that ground truthing is a politically and technically organized agreement about what will count as reality. Synthetic data does not escape this. It is one more place where such agreements are made.
Two words and some questions
So: synthetic data. Two words, and some questions to move forward with:
- How do we teach generative AI to see phenomena?
- What chains of transformations does synthetic data involve?
- How is synthetic data stabilized as evidence?
- How is synthetic data crafted to show certain things?
- How do model cultures and architectures shape synthetic data?
- How does synthetic data circulate and gain authority?
I take up some of these questions in The ontological politics of synthetic data with Saghi Hajisharif and Ericka Johnson, and in The practices and politics of machine learning. I have also argued with Offenhuber at more length in another post.
References
Amann, K., and K. Knorr Cetina. 1988. “The Fixation of (Visual) Evidence.” Human Studies 11 (2): 133–69.
Carlini, Nicholas, Jamie Hayes, Milad Nasr, et al. 2023. “Extracting Training Data from Diffusion Models.” arXiv. https://doi.org/10.48550/arXiv.2301.13188
Drucker, Johanna. 2011. “Humanities Approaches to Graphical Display.” Digital Humanities Quarterly 5 (1).
Gitelman, Lisa, ed. 2013. “Raw Data” Is an Oxymoron. Cambridge, MA: MIT Press.
Jaton, Florian. 2021. The Constitution of Algorithms: Ground-Truthing, Programming, Formulating. Cambridge, MA: MIT Press.
Latour, Bruno. 1995. “The ‘Pédofil’ of Boa Vista.” Common Knowledge 4 (1): 144–87.
Law, John, and Michael Lynch. 1988. “Lists, Field Guides, and the Descriptive Organization of Seeing.” Human Studies 11 (2): 271–303.
Offenhuber, Dietmar. 2025. “Synthetic Data and the Shifting Ground of Truth.” arXiv:2509.13355. https://doi.org/10.48550/arXiv.2509.13355
Vertesi, Janet. 2014. “Drawing as: Distinctions and Disambiguation in Digital Images of Mars.” In Representation in Scientific Practice Revisited, edited by Catelijne Coopmans, Janet Vertesi, Michael Lynch and Steve Woolgar. Cambridge, MA: MIT Press. https://doi.org/10.7551/mitpress/9780262525381.003.0002