Is STS becoming realist again? Reading Dietmar Offenhuber’s “Synthetic Data and the Shifting Ground of Truth”—his talk at 4S 2025 in Seattle, posted on arXiv—I started to wonder. Offenhuber writes that with synthetic data,
ground truth becomes a self-referential affair, in which the labels used as a ground truth repository are themselves synthetic products of a generative model and as such not connected to real-world observations.
In other words: synthetic data has no real-world referent. I disagree, strongly.
Synthetic data without a referent is noise
If there were no referent, why are there women, doctors and educational levels in synthetic datasets? Without a referent, they would not be there. Synthetic data without training data—without a referent—would become what? Noise. That is precisely what a randomly initialized transformer outputs: noise, with the biases of the transformer architecture.
Saying that synthetic data has no referent ignores the theoretical insights and methods that STS has developed over the past decades. What we are dealing with is not a break in the chain of reference, to borrow Bruno Latour’s notion of circulating reference. It is one more step in the many transformations from world to data. So instead of making weirdly realist points about synthetic data, STS should ask how these chains of transformation work: real → data → synthetic data.
Breaking the world, and mimicking it
We need to back up a few steps and ask what computer scientists are doing with synthetic data. Synthetic data is not always made to be faithful to the world. Sometimes it is made to make a model work better—by throwing weird data at it. That is not a representational practice. It is a design practice, where synthetic data is used to shape the performance of a model.
At one point, computer scientists are testing and developing a model: testing edge cases, outliers, weird data. The version of the world they construct is one where extremes are what is sought after. Let’s break the world to test the features of the model! But even this is not without referent or representation. It is not cosmic noise they throw at the model. It is data purposefully designed to push the limits of what a machine learner can predict, generate or steer.
At other points, synthetic data is made to represent the world. But there are purposes here too. We want to mimic things: a text by a specific author, an X-ray image, a population. Here we can discuss what kinds of fidelity are sought after. But saying that the data is disconnected from ground truth is not very analytical. It pays no attention to all the work that goes into making a machine learner output data that actually looks and feels like the world.
Translations and treasons
We can absolutely discuss translations and treasons. What is deemed worthy of representation in synthetic data? Which relationships are preserved? Which aspects of the world are made to travel? And conversely: what is not represented? What transformations of the real data are made? How is it tested?
Perhaps a better question is about mimesis—how the world is represented and imitated through computational transformations. That is something Offenhuber discusses, and should take seriously.
How is synthetic data valued?
There are many questions to ask about synthetic data that don’t fall into a realist trap. We can attend to the tinkering with synthetic data in practice: how it is constructed, and how fidelity is valued differently in different places. As Offenhuber observes, synthetic data is judged by
how well it preserves privacy in a particular application, or the quality of predictions of a model trained with it.
These are questions of valuation and evaluation, rich with social practices, metrics, yardsticks and actors that can be fruitfully untangled. And are they so different from the questions laboratory workers face every day? How well does this dataset represent a patient? What sources of noise are there? Is there enough metadata to describe the dataset?
Offenhuber also writes that “data quality can no longer be considered in relation to reality.” But why would the actors not consider synthetic data in relation to reality? Depending on what the dataset is for, much blood, sweat and tears are shed by the people who create it. Why would they not want it judged in relation to “reality”?
And when he concludes that “data quality is no longer about what the data represent, but rather, what the data do,” I would say: that depends on what the actors want to do. Data quality can be measured in so many ways. One of the most important is whether the data is useful for its purpose.
Ground like coffee
Then, finally, something I really like. Offenhuber borrows it from a data physicalization project at a complexity science workshop in Vienna:
the ground in ground truth no longer refers to a stable reference, but is ground like coffee, boiled and sent through a filter that selectively withholds components.
Yes! And this is exactly why “ground truth” is an epistemic trap. It makes us think like computer scientists. It traps us in a binary opposition between the real and the synthetic. Let’s not throw out the baby with the bathwater: STS has good tools for following chains of transformation. Synthetic data is one more link in the chain—and the interesting questions are about what each link adds, filters out and makes travel.
With Saghi Hajisharif and Ericka Johnson, I have explored some of these questions in The ontological politics of synthetic data.
Reference
Offenhuber, Dietmar. 2025. “Synthetic Data and the Shifting Ground of Truth.” Talk at the Society for Social Studies of Science (4S) 2025, Seattle. arXiv:2509.13355. https://doi.org/10.48550/arXiv.2509.13355