Field Notes

source

A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal

StatusStableUpdatedStands on1 sourceSupportsnotes

Kashmir Hill’s New York Times investigation of 21 August 2022 documented the first publicly reported false positives of Google’s automated CSAM detection. The preserved article is a full-text mirror of the paywalled original; the raw response is at raw copy.

In February 2021, two fathers — “Mark” in San Francisco and “Cassio” in Houston — photographed their toddlers’ genital infections at the request of medical staff during pandemic telemedicine. The photos were automatically backed up to Google Photos, where Google’s classifier flagged them; human reviewers confirmed; Google disabled both men’s accounts across all services (including Mark’s Google Fi phone number) and reported them to the NCMEC CyberTipline as required by 18 U.S.C. § 2258A. Both police departments investigated and concluded no crime occurred; the San Francisco investigator’s report stated the incident “did not meet the elements of a crime.” Google refused to reinstate either account anyway, citing in Mark’s case a six-month-old video of a child in bed with an unclothed woman that reviewers also “considered problematic.”

The article supplies the scale figures this wiki reuses elsewhere: in 2021 Google filed over 600,000 CyberTipline reports and disabled over 270,000 accounts; NCMEC received 29.3 million reports that year, about 80,000 per day, with a staff of 40 analysts prioritizing reports involving potential new victims; the CyberTipline alerted authorities to “over 4,260 potential new child victims,” a count that included both innocent toddlers. It also records Google’s claim that it scans personal images only after a user takes an “affirmative action” — a category that includes automatic cloud backup, the default on Android.

This is the anchoring case study for Automated CSAM detection, CSAM scanning externalizes error costs, and NCMEC CyberTipline. Google’s official account of the same pipeline is in How Google detects and reports CSAM, and the EFF’s analysis in EFF Google scans false accusations. Limitations: the story relies on the two men’s own accounts alongside police documents; Google declined to discuss specifics beyond prepared statements, so the company’s internal review reasoning is known only through those statements.

Built on 1 source (1 external).

Working out connections…