Field Notes

concept

Automated CSAM detection

StatusStableReviewdue UpdatedStands on3 sourcesSupportsnotes

Automated CSAM detection is the practice of scanning user content on online services for child sexual abuse material (CSAM) using perceptual hash matching and machine-learning classifiers, with specialist human review before a platform removes content, suspends accounts, and reports to authorities. It is the load-bearing control point of the US online child-safety regime and the technical model that several jurisdictions, most prominently the EU’s proposed CSA regulation, have considered making mandatory.

Mechanism

Two detection technologies do different jobs. Hash matching computes a perceptual fingerprint of an image or video and compares it against databases of fingerprints of previously confirmed CSAM; Microsoft’s PhotoDNA is the canonical implementation, and NCMEC and the IWF operate hash-sharing services. Hash matching finds known material at scale with negligible false-positive rates, because a match means the content is the same or closely similar to material a human already confirmed.

Classifiers address what hash matching cannot: previously unseen imagery, which is where unidentified, currently endangered victims appear. Google’s Content Safety API (2018) was the first widely deployed CSAM classifier, and Google licenses it along with YouTube’s CSAI Match to other platforms.1 Classification is inherently probabilistic: it generalizes from patterns in confirmed material to novel images, and context — a parent’s medical photo, a child’s own prank video — is invisible to the model. Google states that a specialist team reviews every newly flagged item before any report or enforcement action, and that reviewers are trained by pediatricians to recognize medical-context imagery.1 The Mark and Cassio cases show this human review failing on exactly the category it was trained for,2 which is the commission error Automation bias predicts when a reviewer’s task is to confirm or clear a machine’s flag.

In the United States, 18 U.S.C. § 2258A requires a provider that obtains “actual knowledge” of “facts or circumstances from which there is an apparent violation” of federal CSAM law to report to NCMEC’s CyberTipline, on pain of heavy fines. Section 2258B immunizes providers for those reports, and an SCA exception permits disclosing user content to NCMEC in connection with a report. Crucially, § 2258A imposes a reporting duty but expressly no monitoring duty: scanning is voluntary, yet once a provider scans and finds something, reporting is mandatory.3 This asymmetry shapes the whole ecosystem: providers over-report rather than risk under-reporting, and immunity means a false report costs the provider nothing. CSAM scanning externalizes error costs traces where those costs land instead. Whether compelled scanning would even be constitutional in the US is contested (the Fourth Amendment private-search doctrine and its agency limits), and no US law currently mandates it.3

Scale

The volume numbers matter because they convert any error rate into absolute harm. In 2021 Google alone filed over 600,000 CyberTipline reports and disabled over 270,000 accounts; in the first half of 2022 it passed one million reports with roughly 270,000 suspensions.21 The CyberTipline received 29.3 million reports in 2021 and nearly 36 million in 2023, 85 percent of them from Meta.3 Against that volume, NCMEC alerted authorities to “over 4,260 potential new child victims” in 2021 — a count that included the two innocent toddlers of the NYT cases.2

Error handling and remediation

The documented failure modes are context-blind classification, Automation bias in the review step, account-wide termination (email, photos, documents, phone service in one action), and an appeals process that until late 2022 offered no channel to supply context. After the NYT reporting, Google committed to naming the violation category in suspension notices and to accepting documentation from independent professionals or law enforcement in appeals.1 The 2023 Watkins case shows the reformed process still denying obviously innocent users until a reporter intervened.4 The production false-positive rate of these systems is unpublished everywhere, and What is the production false-positive rate of automated CSAM detection concludes that it is unlikely to become public through the transparency law that appeared to compel it. The reason is worth carrying here, because it also disciplines how any vendor or regulator accuracy claim about this technology should be read. Detection accuracy is measured against a target that is a vanishing fraction of the content scanned, and under that imbalance a headline accuracy figure is close to uninformative: a system that flagged nothing at all would score near the top of the scale. Precision and recall are the statistics that carry information, and they are the ones neither the DSA nor any provider reports. Operationalizing content moderation accuracy in the DSA gives the argument and a worked case where 95 percent accuracy coexists with 58 percent precision and 33 percent recall. An accuracy percentage offered in support of mandated scanning therefore says less than its size suggests.

The encryption boundary

Server-side scanning only reaches content a provider can read, so it is structurally incompatible with end-to-end encryption (E2EE). Meta’s December 2023 default-E2EE rollout on Messenger was followed by a roughly 40 percent one-year drop in NCMEC reports, attributed to lost platform-side detection rather than reduced abuse.5 Proposals to preserve detection under encryption — Client-side scanning, such as Apple’s abandoned 2021 neuralMatch plan — move the scan onto the user’s device and convert a provider-policy question into an architecture question about what a personal device is for. That boundary is the crux of the EU CSA regulation debate covered in EU private communications law.

  1. How Google detects and reports CSAM, Google’s official pipeline description and first-half-2022 figures. ↩↩↩↩

  2. NYT Google CSAM false positives, the Mark and Cassio cases and 2021 scale figures. ↩↩↩

  3. Lawshe v Verizon, on §§ 2258A–2258B, the SCA exception, immunity’s scope, and 2023 report volume. ↩↩↩

  4. NYT child online mistake digital life, the reformed appeals process in practice. ↩

  5. Less harmful social media, on the IWF-attributed post-E2EE reporting drop. ↩

Built on 3 sources (3 external).

Working out connections…