Field Notes

concept

Training data provenance

StatusWorkingUpdatedStands on1 sourceSupportsnotes

Training data provenance is the question of which inputs a trained model’s behaviour derives from, asked after training has already happened. A dataset has provenance in the ordinary sense: records with origins, timestamps, and consent conditions attached. A model trained on that dataset has parameters instead, and the mapping from parameters back to records is not one the training process preserves.

This is a mechanism note rather than a complaint about any company. The property it describes is what makes several otherwise separate disputes structurally identical: erasure requests against trained models, copyright claims over training corpora, and arguments about whether data collected for one purpose ended up serving another.

The asymmetry it creates

Absorption is lossy in a specific and consequential direction. The model retains the statistical regularities of its inputs and discards the index that would let anyone ask which input produced which behaviour.

Three consequences follow, and all three favour the model’s holder.

Deletion becomes ill-defined. Removing a record from a dataset is a well-specified operation. Removing its contribution from a model that has already trained on it is not, short of retraining without it, which for a large model is a capital expense rather than a compliance step. This is what makes data-subject erasure rights awkward to apply to model weights, as distinct from the training corpus, where they apply cleanly.

Denials become unfalsifiable. An organization can state truthfully that a model “is not a copy of, or a means of accessing,” its training data, and that statement is compatible with the data having shaped the model completely. An outside party cannot check it. Nor can they check the opposite claim, which is why accusations in this area tend to be argued from corporate structure and timing rather than from evidence inside the model.

Provenance claims stop being verifiable downstream. A party acquiring or licensing a model inherits whatever its training raises without a reliable way to audit it, which pushes the question from a technical control to a contractual warranty.

Partial techniques, and why they do not close the gap

The property is not absolute. Research on memorization and extraction has shown that models can reproduce specific training examples under some conditions, and work on membership inference asks whether a given record was in a training set at all. Attribution methods that estimate a training example’s influence on a model’s behaviour also exist.

Where the vault currently stands: these techniques establish that provenance is sometimes recoverable in specific, favourable cases. They do not establish that an arbitrary claim about an arbitrary model can be checked by an outside party, which is the question that matters for accountability. The strength and limits of this literature have not been researched in this vault, and this section should not be relied on for a technical or legal argument until they are.

Worked instance

Niantic Spatial is the case that prompted this note. Location scans contributed by Pokemon Go players trained the models behind its visual positioning system; the company later partnered with a defense supplier and stated both that the scans were voluntarily submitted and that its models are not a copy of or a route to those scans. Its partner said it is not using the game’s data while declining to say whether the model it plans to field had been trained on it earlier.

Jeroen van den Hoven of TU Delft, asked to adjudicate, made the provenance point rather than a factual one: once scans are absorbed into a model enriched with many other datasets, identifying which parts came from where is close to impossible.1 That is the whole difficulty in one sentence. The dispute is not open because the reporting is incomplete. It is open because the artefact at its centre does not carry the information that would settle it.

The vault’s other instance is 23andMe, where the data at issue never entered a model and the provenance question stayed tractable. The contrast is the point: what makes the Niantic case unresolvable is not that the data is sensitive but that it has been through training.

Talkie supplies the dataset-side instance. Its pre-1931 corpus exists to test generalization beyond a model’s training data, but the cutoff it claims is a measured property of the corpus — faulty dates and later editorial material can put later knowledge into ostensibly historical records — so even ordinary provenance in the dataset sense is an empirical finding rather than a property of the date filter.

  1. Pokemon Go players unwittingly contributed to tech with military drone uses, Ars Technica, 2026-06-12, reporting van den Hoven’s remarks to Trouw. ↩

Built on 1 source (1 external).

Working out connections…