Field Notes

question

What is the production false-positive rate of automated CSAM detection?

StatusStableReviewdue UpdatedStands on3 sourcesSupportsnotes

Automated CSAM detection systems flag content for human review, account termination, and police reports, but no major platform publishes how often its flags are wrong. The rate is decision-relevant because the same technologies are candidates for mandated deployment: the EU CSA regulation debate (covered in EU private communications law) has featured Commission claims of detection accuracy “significantly above 90 percent” and 88 percent for grooming detection, figures critics note would produce millions of false flags at message scale.1

What is known:

  • Google claims an “incredibly low” false-positive rate for its combined hash-and-classifier pipeline but publishes no number.2
  • A Facebook review of 150 reported accounts found 75 percent had shared the flagged material for “non-malicious” reasons (outrage, poor humor) — not classifier error in the strict sense, but reports that did not correspond to abuse.1
  • LinkedIn confirmed only 31 of 75 hash-matched reports on manual review in H2 2021.1
  • The documented Google false positives (medical photos, children’s own videos) survived both the classifier and trained human review, so the meaningful rate is the post-review false-enforcement rate, which is even less visible.3

The answer, and why it is a finding

This question is closed as unanswerable from evidence this wiki can reach, which is a finding rather than an absence. The obvious explanation — platforms decline to publish — turns out to be the weaker half of it.

The EU created a binding disclosure route. DSA Article 15(1)(e) requires every provider to report “indicators of the accuracy and the possible rate of error of the automated means” it uses for content moderation. On its face that is exactly this question, compelled by law.

It has not produced the number, for two independent reasons.

The compelled metric is uninformative at CSAM prevalence. The Act says “accuracy” and does not define it, specifying neither ground truth nor dataset, so a provider may pick the test set and statistic that flatter it.4 The general principle of a Correction channel applies: an institution whose own outputs supply the only available labels cannot calibrate its accuracy from those outputs, and the DSA’s “accuracy” reporting inherits that structural problem. Literal accuracy is the wrong statistic under class imbalance, and CSAM is the extreme case of class imbalance. Operationalizing content moderation accuracy in the DSA shows a filter scoring about 95 percent accuracy while achieving 58 percent precision and 33 percent recall. At CSAM prevalence the gap is far wider: a classifier that flagged nothing at all would report accuracy indistinguishable from perfect.

Reporting under the provision is documented as empty in practice. The DSA Observatory’s January 2026 assessment finds the requirement “limited in its scope” with “many other problems,” such that information published under it is “essentially meaningless,” and notes that precision and recall appear in no baseline report.5 Its worked attempt to derive Meta’s enforcement precision from Meta’s own disclosures ends without an answer. The DSA Transparency Database does not fill the gap either: over 99.8 percent of statements of reasons cite terms-of-service rather than illegality, and the free-text “decision facts” field is boilerplate in 98 to 99 percent of submissions.6

What remains, and why it is out of reach

Three routes could still produce the figure, and none is public. Article 40 gives vetted researchers access to platform-internal data; Article 42 requires systemic-risk reporting from the largest platforms; and independent audits see internal metrics. Platforms already compute precision and recall internally — that has been documented since a 2019 expert panel commissioned by Meta itself asked for their publication and was refused.5 The number therefore exists and is withheld, rather than being unmeasured.

The definitional problem the earlier version of this note raised stands and compounds the rest: “false positive” can mean a wrong flag, a wrong report, or a wrong enforcement action, and any published figure would have to say which. The documented Google cases survived both classifier and human review, so the decision-relevant quantity is the post-review false-enforcement rate, which is the least visible of the three.

  • Watch for an Article 40 study or Article 42 systemic-risk report that publishes precision for a child-safety classifier; that would reopen this question with an actual number.

  1. EFF Google scans false accusations (Joe Mullin, “Google’s Scans of Private Photos Led to False Accusations of Child Abuse,” EFF Deeplinks, 22 August 2022); saved copy. ↩↩↩

  2. How Google detects and reports CSAM (Google, “How we detect, remove and report child sexual abuse material,” 28 October 2022); saved copy. The company’s own account of its pipeline, published two months after Hill’s story, and the only description of it the vault holds. ↩

  3. NYT Google CSAM false positives (Kashmir Hill, “A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal,” The New York Times, 21 August 2022); saved copy. ↩

  4. Operationalizing content moderation accuracy in the DSA (Johnny Tian-Zheng Wei, Frederike Zufall, and Robin Jia, “Operationalizing content moderation ‘accuracy’ in the Digital Services Act,” arXiv:2305.09601, 2023); saved abstract page. ↩

  5. DSA Observatory, “The Missing Metrics in DSA Content Moderation Transparency”, 8 January 2026. The repository downloader could not reach the host, so this is cited from the live page rather than a local snapshot. ↩↩

  6. Kaushal, van de Kerkhof, Goanță, Spanakis, and Iamnitchi, “Automated Transparency: A Legal and Empirical Analysis of the Digital Services Act Transparency Database”, 2024, analysing 131 million statements of reasons. ↩

Built on 3 sources (3 external).

Working out connections…