Case study · 2026-07-12

Population-scale content scanning, measured

Two numbers decide whether a population-scale detector helps or harms: how often it flags the innocent, and how easily a sender slips past. Both are usually asserted, not measured. I measured them on the CSAM perceptual-hash example, on a deployed hash plus open ones, and had four adversarial passes try to break my own numbers first. Three of them succeeded.

Matteo Panzeri · AI-for-security research

Written with AI assistance and human primary-source verification; the adversarial audits referenced were independent adversarial LLM passes, not third-party review. Every legislative and technical claim carries a primary source; all code, per-image data (CSV), dependency versions and fixed seeds are published. Perceptual-hash experiments were run on benign standard test images only.

The claim

When a machine-learning classifier or perceptual hash is deployed to scan an entire population’s private messages for child-sexual-abuse material (CSAM), policy-violating content, or jailbreaks, two questions decide whether it helps or harms: how often it flags the innocent, and how easily a motivated sender slips past. As the recent EU “chat control” debate (dated note at the end) shows, both are loudly mis-stated from both directions. This piece models the first and measures its binding input, then measures the second, on the CSAM-scanning example, with open code. The honest answer turns on two axes people leave unstated: the flag-rate scales with prevalence and population scale, and evasion scales with attacker effort and the specific hash. Neither “it works” nor “it’s trivially broken” survives contact with them.

1 · Base-rate collapse: even a good detector floods the innocent

Detection accuracy is not what matters at scale; the base rate is. Let PPV be the probability that a flagged message is genuinely target material. By Bayes, PPV = (π·TPR) / (π·TPR + (1−π)·FPR), where π is prevalence and FPR the per-message false-positive rate. With a generous true-positive rate (TPR = 0.9), the model prints where PPV lives and where it reverses:

PPV = share of flags that are genuinely target material, TPR = 0.9. Output of base_rate_model.py.

The binding premise is the per-message FPR. In the believed regime (FPR of roughly 1e−3 to 1e−4) PPV stays low: at π = 1e−5 and FPR 1e−3, PPV ≈ 0.9%, i.e. roughly 99 in 100 flags are innocent people. The result reverses only at FPR ≤ 1e−6 and higher prevalence, so it depends on the (π, FPR) pair and is shown, not asserted.

Is that FPR regime real, or convenient? To stop assuming it, I measured a per-comparison false-match rate directly: over all 124,750 pairs of 500 benign natural images (BSDS500), how often do two different images collide within a hash’s deployed match threshold? The pairs are not independent (each image is in 499 of them, which a naive interval ignores), so the confidence intervals below are image-level bootstraps:

Per-comparison false-match rate on benign pairs. Output of perceptual_hash_boundary_v2.py. Open-hash intervals are image-level bootstraps; PDQ’s zero-count bound is a rule-of-3 estimate (~3e−5) that assumes independence and so under-states the true upper bound.

This empirically lower-bounds the model’s FPR input for the open hashes: dHash sits solidly in the base-rate-collapse regime (≥ 1e−4 even at its CI floor); pHash’s interval straddles 1e−4, so I can’t claim it is squarely inside. Two honest caveats keep this from being oversold. First, this is a per-comparison rate; the model’s per-message FPR is roughly this times the size of the known-bad database (a benign image is checked against every entry), so the measurement is a floor, not the deployed rate. Second, the one deployed hash tested, PDQ, produced zero false matches, so its per-comparison rate is below what 124,750 pairs can resolve. The flooding argument is therefore strongest for the open 64-bit class and, for PDQ, rests on database-scale compounding I did not measure. (For scale: Farid, PhotoDNA’s co-creator, has cited a ~1-in-50-billion false-match rate in 2019, a figure whose derivation was never published and never independently verified.)

Absolute harm, unlike PPV, is not scale-invariant: at, conservatively, ten billion private messages a day (an order-of-magnitude assumption; WhatsApp alone reports ~100B), even a 1e−4 rate flags on the order of a million innocent messages daily. An independent real-world anchor, precisely labeled: in Ireland (2020), of 4,192 NCMEC referrals, 852 (20.3%) were marked as CAM, 409 (9.7%) actionable, and 265 (6.3%) completed cases, all from a pipeline already enriched by hash-matching plus human triage, so a later stage than the model’s raw PPV and not a validation of it (“completed” is the Garda’s own case-marking, not prosecutions, which per the ICCL were not retained).

2 · The evasion boundary: a fair test, on a deployed hash

The academic record looks contradictory. Jain, Crețu & de Montjoye (USENIX Security 2022) report black-box attacks evading perceptual-hash detection on >99.9% of images. Madden, Bhavsar, Dorje & Li (NeurIPS 2024 workshop, arXiv:2406.00918) find, “contrary to existing literature … these [algorithms] demonstrate significant robustness,” attributing high evasion figures to “the random hash variations characteristic of” perceptual hashes. Both are verified to primary source, and both can hold, because “evasion rate” is under-specified.

I measured it on two open hashes (pHash, dHash; 64-bit) and, crucially, on PDQ, Meta’s 256-bit hash actually deployed for content matching (open-source; match threshold ≤31). The essential control is a query-budget-matched random baseline: an attacker making N guided queries is compared against N random ones. Over 100 benign images, five seeds, one-sided paired tests with Holm correction, the result is hash-dependent:

• dHash (a difference hash) is genuinely attackable: a directed advantage over matched random survives correction across all five seeds even at an invisible budget, grows with query count (emerging by ~150 queries and rising to ~+1.7 bits), and becomes operationally real at a visible budget (19 of 100 images pushed past the threshold vs 5 for random). This is the Jain direction.
• pHash (a DCT hash) shows no advantage at any budget (the directed attack is, if anything, worse than random search), and this holds even when I strengthen the attack with eight random restarts (0 of 100 evade). Genuine robustness to a simple attacker. The Madden direction.
• PDQ (the deployed DCT hash) is the honest middle. The simple attack showed no advantage; a stronger restart attack does find a real directed signal (p < 1e−4), so PDQ’s apparent robustness to the weak attack was partly an artifact of a weak optimizer, not the hash. But the signal is only ~+1.6 bits against a 31-bit threshold margin, so 0 of 100 images evade at an invisible budget. Operationally Madden; mechanistically not zero-signal.

What about the “hash is naturally unstable” explanation? Decomposing benign transforms (JPEG-50, 0.9× resize, ×1.1 brightness, a 2° rotation), the instability is almost entirely the rotation: JPEG/resize/brightness move every hash by under a bit or two, while a 2° rotation moves PDQ ~36 bits (past its threshold for 74 of 100 images). An adversarial pass flagged that this could be an artifact of black corners from the rotation fill; re-running with edge-extension instead cut it (from 43 to 36 bits) but the effect persisted, so PDQ’s small-angle rotation sensitivity is real, and it is not covered by the dihedral (90°) database augmentation deployments use. That is a genuine residual gap, and it means less of the “evasion” at invisible budgets can be waved away as natural noise than the naive story assumes.

Limits, stated as the study’s spine

• The attack is a simple greedy search (with restarts), a lower bound. A stronger attack shifts the curves left; “a simple attacker, fairly compared, does not evade the DCT hashes at an invisible budget” is the claim, not “the hash is robust.”
• The “DCT resists / difference-hash is attackable” split is a correlation across three algorithms (two DCT, one difference), confounded with bit-length (PDQ is 256-bit), not a validated causal mechanism. Two DCT hashes cannot establish a class property.
• pHash/dHash are open; the deployed hash tested is PDQ. PhotoDNA and NeuralHash are not tested (PhotoDNA is not even Hamming-comparable, and NeuralHash is excluded by risk posture), and nothing transfers to those two.
• The false-match rate is per-comparison, not per-message; 100 images for evasion, 124,750 pairs for FPR; PDQ’s true low rate is below resolution.

3 · What the data do and do not show

A “working” detector still floods the innocent at population scale, and the sensational evasion figures require an unstated strong-attacker, high-query, specific-hash setting. The data do not show “detectors are broken,” and do not crown Jain over Madden: the verdict is genuinely split by hash. The base-rate structure generalizes to any population-scale content-scanning classifier (moderation, jailbreak and safety filters share the low-prevalence, high-volume shape); the evasion result is perceptual-hash-specific and does not. The contribution is methodological: a reproducible way to force the question into its real variables (prevalence, scale, query budget, hash architecture, perturbation visibility) and watch the answer move.

How I red-teamed my own numbers

Most analyses show you their conclusions. This one shows the process, because on this topic the process caught me being wrong repeatedly before anything was published, and which errors a method catches is more informative than which conclusions it reaches. Every claim was verified against a primary source in both directions, and each artifact was handed to independent adversarial passes whose job was to break it. Four such passes (statistics, overclaim, code-correctness, factual) ran on the measurement itself; here are the load-bearing catches.

• My headline was refuted by my own table. An early draft asserted the base-rate result “does not depend on any contested number”; the model’s own printout showed PPV climbing above 90% in one corner of the grid. The result is now stated conditionally, with the model printing exactly where it reverses.
• A hidden confound reversed a provisional finding. My first evasion read compared a 300-query attack against a 6-query random baseline (best-of-300 vs worst-of-6). Under a query-budget-matched baseline with corrected statistics, the advantage vanished for the DCT hashes; I then turned the residual limitation into the study by sweeping attack strength.
• My real-world anchor was mislabeled. I had cited Ireland’s 20.3% as an “actionable/PPV” rate; it is the share marked as abuse material, from a later, enriched pipeline stage, not comparable to the model’s per-message PPV. It now appears as an independent datapoint, explicitly not a validation.
• The strengthening pass caught three more. When I added the deployed hash and a larger corpus, the adversarial passes found: my false-match confidence intervals assumed 124,750 independent trials when the pairs share images (fixed with an image-level bootstrap, which widened them enough that pHash’s interval now straddles 1e−4); my “rotation breaks the hash” figure was partly a black-corner fill artifact (re-measured with edge-extension, the effect shrank but survived); and my “DCT hashes resist” null was partly an artifact of a weak optimizer, since a stronger restart attack found a real signal on PDQ. Each correction made the write-up less tidy and more true.

None of these would have been caught by a reader; several contradicted my own files. Publishing the corrected numbers is table stakes. Publishing the record of how the corrections happened is the actual demonstration of the habit a security or AI-safety role runs on: red-team your own preferred conclusion, and report the result that weakens it.

Reproducibility

Code, per-image data (CSV), dependency versions and fixed seeds are published: base_rate_model.py, perceptual_hash_boundary_v2.py (PDQ via the open-source pdqhash; corpus BSDS500), attack_budget_sweep_v2.py, strong_attack_check.py (with CSVs). Every legislative and technical claim is footnoted to a primary source.

Sources

• Jain, Crețu, de Montjoye, Adversarial Detection Avoidance Attacks, USENIX Security 2022.
• Madden, Bhavsar, Dorje, Li, arXiv:2406.00918 (NeurIPS 2024 AdvML workshop); quotes and venue verified.
• PDQ: Meta ThreatExchange (github.com/facebook/ThreatExchange/tree/main/pdq): 256-bit hash, documented match threshold ≤31; the pdqhash Python package.
• Struppek et al., Learning to Break Deep Perceptual Hashing, FAccT 2022, arXiv:2111.06628.
• Ireland 2020 figures: Irish Council for Civil Liberties, citing An Garda Síochána: 4,192 / 852 / 409 / 265 / 471; prosecution figures not retained.
• PhotoDNA ~1-in-50-billion: attributed to Hany Farid (co-creator), 2019; derivation unpublished, never independently verified.
• European Parliament press release ref 20260706IPR46318 (the 9 July second-reading vote).