Today we had the great pleasure of hosting Benjamin Jacobsen (York University) for a talk at the Intersynth Lab. He is the PI of an ERC project on synthetic data that promises to be really interesting, and in his talk he zoomed in on the big AI labs and how they use synthetic data to train models. These are my notes from his talk
Benjamin’s argument
Benjamin’s main question in the talk was what happens when AI models are recursively trained on synthetic data? He wanted to focus not on its future promises or risks, but on how synthetic data is already shaping the present landscape of generative AI. It seems to have become an inescapable part of the current generative AI development pipeline. Theoretically, Benjamin drew on Deleuze’s notion of the fold—more on that below. He also referred back to some of the work I have done on algorithms as folding.
An important point is that the history of synthetic data is also the history of the technologies we have used to generate it. Benjamin focused on AI-generated synthetic data, and according to him, synthetic data made with statistical modeling might be very different. So one of his questions are: how can we generalize from AI-generated synthetic data to other methods of generating data?
How and where synthetic data is used
He started by outlining where and how synthetic data is used: to train, test, fine-tune and evaluate machine learning models, to experiment with them. Synthetic data are also used in healthcare, political polling, finance, retail, and human trafficking. It is generated in very different ways: with mathematical or statistical models, through simulation or 3-D rendering engines, or with machine learning models and algorithms. The big AI labs have published a lot of work on synthetic data: For instance, how it can be used to reduce the sycophancy of models. Another thing it can be used for is tailoring the models to align better with the ethical values you like by using synthetic data. It is also used to make models more robust against hallucinations. And this has been happening for a few years now; it is not a new thing.
Whose worlds get represented?
A particularly interesting quote that Benjamin took up was from: Hooker (2025): “We are now able to intentionally skew the distribution towards what we hope to represent, rather than accepting a random sample of the world as it is.”
One example of this was when he discussed how the model DALL-E 3 was trained to understand users’ prompts better by creating simulated prompts for images. The model that generated prompts was fed an image and instructed to write a very detailed prompt describing it. This made the image model better at generating images from prompts. The problem, of course, was that once the model was taught to respond to very detailed prompts, it became worse at understanding less detailed prompts, like the ones humans write. He also went through the famous paper on model collapse. A model is retrained again and again on synthetic data, and the result is that the tails of the distribution are cut off—which we have also written about in The ontological politics of synthetic data—and the model becomes less and less able to account for diversity.
In his talk, Benjamin posed a question that is particularly salient in my view: who gets to decide what kind of world our AI models represent? This ties, of course, to Elon Musk training his models on the values he likes, while other models, such as Anthropic’s, are trained on other values. A great question that Benjamin asked was: if we don’t want to accept the randomness of the world, who will be able to decide which world is the correct one for training AI models?
And of course, all the big labs are working to fine-tune their models towards the values they want to see in them. This is a question of ontology, of what reality becomes represented, but also a question of politics: which politics becomes represented in these models that will become infrastructures of knowledge making?
What does it mean for a model to work?
At this point a question popped up in my head: what does ”working” mean? What is a ”successful” model? In a world of incessant benchmarking, where labs use various metrics of excellence for their models, it becomes very interesting to look at what ”working,” ”success,” or ”good” means. This ties to questions in valuation studies about how these metrics are constructed: turning these evaluations into verbs, and looking at the processes by which models are constructed as excellent. Svea Kiesewetter has also been thinking about and working on this, for instance in her talk at EASST 2026 in Kraków.
Another thing that popped up in my head during the talk was the question of ”working,’” as the construction of truth, or of verisimilitude. For some things it is quite easy to find out whether a model is working. In coding, for instance, there are automatic ways to test whether code works. But it becomes much more complex when we are dealing with humans and human lives, not to mention the complexities of biology, which we hardly understand today.
The fold
Theoretically, Benjamin finished by connecting to Deleuze’s work on the fold. He also referenced our work on folding and quoted a sentence from the article Algorithms as Folding: “we can gain a better understanding of how they become part of ordering the world: sometimes superimposing things that might seem distant and sometimes tearing apart things that might seem close.” Benjamin then discussed his interest in terms such as mixes, mixtures, proportions and blends, which point to the folding of different types of data.
To me, this also brought to mind Nanna Thylstrup and Kristian Bondo Hansen’s work on stack bricolage. I have written about this layering before, in a post on agency and power in AI assemblages, with the xkcd comic about all of modern digital infrastructure resting on a project that some ”random person in Nebraska has been thanklessly maintaining since 2003.” The folding of infrastructures and digital things raising questions about the traveling of ontologies and epistemologies in digital infrastructures.
Folding models into models
Here Benjamin started talking about the politics of folding models into each other, which I have also been thinking and writing about in relation to algorithms. His example was that Nvidia built a model on top of Llama 3. The Nvidia model was then fine-tuned on a mix of real and synthetic data, and perhaps also on data from Chinese models. So here the concept of the fold that Benjamin started out thinking with becomes important.
I think Karen Barad’s work on the agential cut is also relevant here—which is something my colleague Ericka Johnson also has been thinking about: Where do data and model start and end? It seems that we are drawing arbitrary lines between the model and the data when these are inextricably folded together. And perhaps more importantly, what are the politics involved in folding models into other models into other models? What representations become folded into this complex network of models? What ontologies and epistemologies become part of other models? This reminds me of Bilel Benbouzid’s work on how earthquake prediction algorithms have been repurposed for predictive policing.
Conclusion
Benjamin’s conclusion is that we need to attend to the synthetic conditions of generative AI models. We need to pay attention to the fact that they are always mixed and folded with other things, and that the underlying distributions are already a mixture and a folding of real and synthetic data.
And here we are back to what annoyed me in Dietmar Offenhuber’s paper, which I have blogged about, where he talks about real and synthetic data as if they existed as separate, pure categories.
References
Barad, Karen. 2007. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning. Durham: Duke University Press. https://doi.org/10.1215/9780822388128
Benbouzid, Bilel. 2017. “Des crimes et des séismes. La police prédictive entre science, technique et divination.” Réseaux 206 (6): 95–123. https://doi.org/10.3917/res.206.0095
Bercovich, Akhiad, Itay Levy, Izik Golan, et al. 2025. “Llama-Nemotron: Efficient Reasoning Models.” arXiv:2505.00949. https://doi.org/10.48550/arXiv.2505.00949
Betker, James, Gabriel Goh, Li Jing, Tim Brooks, et al. 2023. “Improving Image Generation with Better Captions.” OpenAI. https://cdn.openai.com/papers/dall-e-3.pdf
Deleuze, Gilles. 1993. The Fold: Leibniz and the Baroque. Translated by Tom Conley. Minneapolis: University of Minnesota Press.
Hansen, Kristian Bondo, and Nanna Thylstrup. 2024. “Stack Bricolage and Infrastructural Impermanence in Financial Machine-Learning Modelling.” Journal of Cultural Economy 17 (1): 20–38. https://doi.org/10.1080/17530350.2023.2229347
InterSynth Lab. https://intersynth.ai
Jacobsen, Benjamin. University of York. https://www.york.ac.uk/sociology/people/benjamin-jacobsen/
Johnson, Ericka. https://www.erickajohnson.se
Kiesewetter, Svea, Francis Lee, and Ericka Johnson. 2026. “Reassembling the Local in Synthetic Data Infrastructures.” Paper presented at EASST 2026, Kraków, 8 September. https://nomadit.co.uk/conference/easst2026/paper/100942
Latour, Bruno. 2002. “Morality and Technology: The End of the Means.” Translated by Couze Venn. Theory, Culture & Society 19 (5–6): 247–260. https://doi.org/10.1177/026327602761899246
Lee, Francis. 2024. “Who Gets to Choose? Agency and Power in AI Assemblages.” Blog post, 28 November. https://francislee.org/blog/2024-11-28-agency-and-power-in-ai-assemblages.html
Lee, Francis. 2026. “Don’t Throw Out the Baby with the Bathwater: Synthetic Data and the Real.” Blog post, 29 January. https://francislee.org/blog/2026-01-29-synthetic-data-and-the-real.html
Lee, Francis, Jess Bier, Jeffrey Christensen, Lukas Engelmann, Claes-Fredrik Helgesson, and Robin Williams. 2019. “Algorithms as Folding: Reframing the Analytical Focus.” Big Data & Society 6 (2). https://doi.org/10.1177/2053951719863819
Lee, Francis, Saghi Hajisharif, and Ericka Johnson. 2025. “The Ontological Politics of Synthetic Data: Normalities, Outliers, and Intersectional Hallucinations.” Big Data & Society 12 (2). https://doi.org/10.1177/20539517251318289
Munroe, Randall. 2020. “Dependency.” xkcd 2347. https://xkcd.com/2347/
Offenhuber, Dietmar. 2025. “Synthetic Data and the Shifting Ground of Truth.” arXiv:2509.13355. https://doi.org/10.48550/arXiv.2509.13355
Shumailov, Ilia, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. “AI Models Collapse When Trained on Recursively Generated Data.” Nature 631 (8022): 755–759. https://doi.org/10.1038/s41586-024-07566-y
Valuation Studies. https://valuationstudies.liu.se/