Field Notes

synthesis

CSAM scanning externalizes false-positive costs onto users

StatusStableReviewdue UpdatedStands on4 sourcesSupportsnotes

Automated CSAM detection at platform scale produces false positives whose costs fall almost entirely on the wrongly flagged user, because the legal and institutional design shields every other actor in the pipeline. This note argues that the allocation is structural, not a string of individual company failures, and that any proposal to extend scanning — such as EU detection orders — inherits the same cost allocation unless it designs remediation in from the start.

The observed pattern

The documented cases share a shape. An automated flag, human-confirmed, triggers account-wide termination and a mandatory report to the NCMEC CyberTipline; police investigate with full access to the user’s stored data; police clear the user; the platform declines to reverse its own judgment. Mark and Cassio, the two fathers of the 2022 NYT investigation, lost decade-old accounts, family photos, and in Mark’s case his phone number and the two-factor channel for the rest of his digital life, even after the San Francisco investigator wrote that “no crime occurred.” Watkins, in the 2023 follow-up, was flagged for her seven-year-old’s own prank video, exhausted the reformed appeals process, and was restored only after a reporter inquired. A Kansas user, Lawshe, was arrested on an auto-filed report over legal adult pornography tagged “unconfirmed CSAM.”1

Why the costs land where they do

Each layer of the stack is insulated. The classifier’s operator faces no liability for a wrong flag; the platform is immunized for reports by § 2258B and fined for silence under § 2258A, so its dominant strategy is to report and let others sort it out;1 NCMEC is a triage bottleneck that must treat incoming reports as leads; police act on the lead with intrusive but lawful process. The user is the only actor who cannot pass the cost downstream. Pfefferkorn’s summary is that these false positives “cost the provider nothing but cost the user dearly.”1 Criminal-defense practitioners describe the same asymmetry from the other end of the pipeline: an Ohio defense firm notes that a closed investigation still leaves the accused with the investigation and its reputational residue, because clearing someone is not the same as undoing the accusation. Even the remedy Google publicized in 2022 — clearer notices and context-accepting appeals — changed the process without changing outcomes absent media leverage, as Watkins’s case shows.

Three mechanisms make false positives irreducible rather than merely unpolished. First, classifiers cannot see context: whether an image of a child’s body is medical documentation, a family snapshot, or abuse material is a fact about the world outside the pixels. Second, the enforcement unit is the account, not the content, so one flag detonates a user’s entire digital life across mail, files, photos, and telephony. Third, Automation bias and the over-report incentive push in the same direction, so review confirms rather than corrects — Google’s pediatrician-trained reviewers confirmed both fathers’ medical photos, and its later review of Mark’s account recast a family video as grounds for standing by the decision.2 Comparative data points the same way: Facebook found 75 percent of a reviewed sample of reported accounts had shared flagged images for “non-malicious” reasons, and LinkedIn confirmed only 31 of 75 hash-matched reports on manual review.3

Counterweight

The system’s defenders hold — with real justification — that under-detection has worse victims than over-detection: a missed report can leave a child in an abusive situation, and CyberTipline reports do rescue children. The Stanford report’s interviewees, including civil-society critics, treat the pipeline as worth fixing rather than abandoning.4 The synthesis here is not that scanning should stop; it is that the current design prices its errors at zero for everyone except users, so nothing in the system optimizes for reducing them. Pfefferkorn’s analysis of Lawshe notes the same logic from the provider side: immunity is itself the regulation, and puncturing it case-by-case would swamp the pipeline.1

Implications

  • Proposals to extend scanning into new channels (encrypted messaging under the EU CSA regulation, covered in EU private communications law) multiply the flagged population by orders of magnitude; the remediation question is therefore a first-order design requirement, not an afterthought. A detection mandate without a remediation mandate replicates the US allocation at larger scale.
  • Remediation design has identifiable levers the cases point to: decoupling content removal from account-wide termination, a duty to restore demonstrably cleared users, publishing false-positive rates, currently unknown for every major platform (What is the production false-positive rate of automated CSAM detection), and routing cleared-by-police outcomes back into platform decisions.
  • For users, the cases are a concrete instance of account-concentration risk: a single provider holding mail, photos, documents, phone service, and login federation turns one moderation error into total digital exile.

  1. Lawshe v Verizon (Riana Pfefferkorn, “Verizon and Its Cloud Vendor Must Face Lawsuit for Reporting ‘CSAM’ That Wasn’t,” guest post on Eric Goldman’s Technology & Marketing Law Blog, March 2025, updated February 2026); saved copy. The first suit against providers over a false CSAM report to survive a motion to dismiss. ↩↩↩↩

  2. NYT Google CSAM false positives (Kashmir Hill, “A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal,” The New York Times, 21 August 2022); saved copy. The first publicly reported false positives of Google’s automated detection. ↩

  3. EFF Google scans false accusations (Joe Mullin, “Google’s Scans of Private Photos Led to False Accusations of Child Abuse,” EFF Deeplinks, 22 August 2022); saved copy. An advocacy analysis responding to the two cases Hill reported. ↩

  4. Online child safety ecosystem (Grossman, Pfefferkorn, Thiel, Shah, DiResta, Perrino, Cryst, Stamos, and Hancock, The Strengths and Weaknesses of the Online Child Safety Ecosystem, Stanford Internet Observatory, April 2024); saved PDF. Traces the US CyberTipline pipeline from platform report to potential prosecution. ↩

Built on 4 sources (4 external).

Working out connections…