Notebook · 2026-06-03

Measuring whether my method beats a naive frontier-model call

On crypto a method won. On authz, a static proxy of it did not, and the runtime test recovered the case it missed. Telling those two apart is the point.

Matteo Panzeri · AI-for-security research

TL;DR

Models now find vulnerabilities at enormous scale, so raw discovery is commoditizing. The scarce, durable skill is no longer finding a bug; it is measuring where the model over-claims, and whether a given method actually beats a plain model call. I built a small, open, model-agnostic benchmark and scorer for that question (sota_bench), with a pre-registered loop that re-runs against each new frontier release. The baseline is a single naive call to a frontier model I run; I make no claim against closed systems I cannot run, such as Mythos. On its first runs it told me two different things about two of my methods. On crypto, a forcing-function method beat the model. On a small authorization benchmark, a static-prompt proxy of my sibling-guard method did not beat a naive single call. That proxy omits the method's actual mechanism, a runtime check, so the result scopes where the value is not (static skepticism); it does not falsify the method. And when I ran the method's actual runtime mechanism on a small subset, it recovered the exact case static skepticism had missed. The instrument's job is precisely this: to stop me both from over-claiming "my method wins" and from unfairly concluding it fails.

Why measurement, not discovery

Anthropic's own published figures make the case. Their autonomous system reported 23,019 candidate findings. A hand-assessed slice of about 1,900 ran at 90.8% true-positive, and 1,596 were disclosed as vulnerabilities. Only a small fraction reached a CVE or GHSA, and the model matched external-firm severity exactly 58.7% of the time (94.4% within one band). The throughput bottleneck has moved from detection to triage, validation, and calibration, and the published literature on LLM severity calibration and false-positive rates is recent and unconsolidated. That is the gap: not another finder, but an instrument that measures which findings are real, how severe they actually are, and whether a method earns its keep against a baseline that improves every release.

The harness

sota_bench is a stdlib-only Python package with a non-LLM scorer (no model-as-judge, which is too fragile for a validation layer), reusing established metrics so results are comparable rather than bespoke:

• detection: precision, recall, false-positive-rate, OWASP-Benchmark-style TP/FP/TN/FN and Youden's J, PrimeVul's VD-S (false-negative rate at a fixed false-positive cap), and pairwise positive/negative accuracy;
• severity: a signed, both-ways calibration error versus an honest CVSS v3.1 band, reporting inflation and deflation separately rather than as one absolute number;
• a labeled positive/negative corpus pinned to repo@commit, with each negative carrying the runtime check that exonerates it;
• a SOTA-validation loop: pin a named naive-single-model baseline, re-run on each frontier release, and publish the signed "method minus naive" delta. The corpus is dated and hand-adjudicated, which is the part a newer model cannot retroactively author.

The design borrows freely: discover-then-validate, patched-counterpart negatives, the "inconsistency equals bug" oracle, and baseline pinning are all prior art. The contribution is the combination, the two substrates, and the loop.

Experiment 1: decode-completeness (crypto). The method helped.

On a finished, public DeFi audit contest (Monetrix, where the real finding was independently found by dozens of auditors), a naive multi-axis blind pass over the accounting code reached the exact decisive site, recovered the external precompile's full field layout, and then confidently concluded there was no bug (it pursued a plausible-but-wrong alternative). A method-armed pass that does one forcing-function step, "for every external data source, enumerate every field of its real schema and flag any value-reducing field the decoder drops," re-derived the real finding cleanly. In the pre-registered baseline run the split was 0/3 naive versus 2/2 method on the held-out medium-severity bug (an earlier three-reviewer calibration independently re-derived it 3/3), with the other five external reads correctly exonerated. Here the method beat the model. The gap was not knowledge, the naive pass had the data; it was the discipline of refusing "the decoded field is correct" until completeness is checked.

Experiment 2: agent tool-dispatch authorization-confusion. A static proxy of the method lost.

authz_v1 is a labeled set in the class where an application enforces an authorization guard on its REST path but omits it on the agent or tool-calling dispatch path (CWE-862/863, confused-deputy). The positives are coordinated disclosures still under embargo, so they are withheld from this public version and will be added once their advisories publish; the negatives are by-design-secure agent surfaces, including two near-miss cases that look flaggable statically but are cleared by a runtime check. I ran a blind, code-only comparison (each pass reads only the source at repo@commit, forbidden from looking up any advisory): a naive single call versus a call armed with my sibling-guard / runtime-gating method.

Pinned baseline, 2026-06-03, over the full 16-item labeled set (6 positive / 10 negative). The signed method-minus-naive delta is recall −0.167, precision −0.20, FP-rate +0.10, Youden J −0.267: the static proxy is worse on every detection axis. The public sota_bench repo currently ships only the by-design-secure negatives; the positive items and two secure twins are withheld pending coordinated disclosure, so these exact rates are not reproducible from the public slice yet.

The static proxy was strictly worse. Read statically, its extra skepticism talked itself out of a real finding (one positive case, a false negative) and over-flagged a by-design shared-capability surface (AnythingLLM, a false positive). The naive frontier model is already strong at spotting a missing authorization check from code; bolting "be more skeptical, look for the sibling guard" onto a static read did not help, it hurt. Crucially, this is the method stripped to a static prompt: its actual discriminator is a runtime check (is the caller denied on the sibling path yet still reaching the action via dispatch?), and that step is what cleared these same by-design cases and confirmed the real ones when the advisories were originally filed. A static read cannot run it, so what underperformed here is static skepticism, not the method.

Experiment 2b: the runtime oracle, the fair test of the method

The static proxy is not the method; the method's discriminator is the runtime check, so I ran it. Re-running the original dynamic PoC harnesses against pinned affected versions, with fresh CSPRNG sentinels minted during the run, the runtime method matched ground truth on 4 of 4 findings in the subset I could stand up, and its one decisive move was the exact case the static proxy got wrong. On that case the static proxy had cleared a real vuln to "secure" because the agent route did carry an authorization decorator, but it was scoped too narrowly, so the dispatch only looked gated. The runtime oracle flipped it back to vuln: a normal team member, denied on the REST entitlement path, nonetheless reached owner-private content through the dispatch sink, returning two freshly-minted sentinels with no authorization denial. A second case stayed vuln on a live re-run, and the runtime oracle correctly cleared a by-design-secure runtime path (the caller identity is bound at dispatch), so it did not over-flag. The specific targets are coordinated disclosures still under embargo, named here only once their advisories publish. This is the method's real mechanism recovering precisely what static skepticism missed.

Honest coverage: only 2 of the 4 were truly live-reran; the other two verdicts rest on recorded evidence (the live re-run was not completed, with no new sentinel minted). And this subset does not include the static proxy's other error, the false positive on a by-design AnythingLLM surface, so the runtime method's false-positive discrimination is only partially shown.

What the two results mean together

The value of a method is substrate-specific and stage-specific. Decode-completeness, a forcing function that makes the model do work it otherwise skips, helped. Authorization sibling-guard reasoning, applied as static skepticism, did not beat a model that already reads authorization well. That run tested the method as a static prompt; its actual designed edge is a runtime check, which I ran separately in Experiment 2b, and on a small subset it recovered exactly the case static skepticism missed. The honest reading: my authz method has no demonstrated edge over a naive call for static detection, and I would be wrong to claim otherwise; its value is the runtime verifier, which on a tiny subset corrected the static proxy's one error. I would be equally wrong to call this the method failing, because the experiment never ran the method's mechanism. The precise, defensible claim is narrow: static skepticism without a runtime verifier is not an edge; whether the runtime verifier is one is the next experiment.

The actual contribution

The point is not which method won. The point is that the measurement let me find out, and scope the claim precisely. Instead of over-claiming "my method wins" or unfairly concluding it fails, it told me where a method's value lives: a forcing-function helped; static skepticism without a runtime verifier did not. It also flagged that my authz method's actual mechanism is still untested. That instrument, an open, model-agnostic benchmark with a non-LLM scorer and a re-run-every-release delta loop, gets more useful as models improve, because someone has to measure whether and where each new model is actually better, and whether yesterday's method still earns its place. That instrument is durable and compounding, but it is the supporting exhibit, not the asset. The asset is the specialization shown through real disclosures plus the repeatable sibling-diff and incomplete-fix method; the measurement is what proves those methods are held to a baseline.

Limitations (read these)

• A single run, a single model, static code only, over a small labeled set whose positive items are withheld pending coordinated disclosure. This shows a forcing-function method helped on one substrate while a static proxy of the sibling-guard method underperformed on the other; it is not a calibrated rate.
• The runtime fp-killer was exercised (Experiment 2b) but only on a 4-finding subset, of which 2 were live re-runs and 2 rest on recorded evidence. It corrected the static proxy's one false negative and cleared a by-design-secure path, but the subset excludes the AnythingLLM false positive, so the runtime method's FP-discrimination is only partially shown.
• The decode-completeness "no prior benchmark" claim is low-confidence pending a dedicated null-search.
• Experiment 1 is a single retro-find of a known contest bug (independently found by dozens of auditors): it shows the method recovers a finding the naive pass missed, not that it first-finds novel bugs.
• The corpus is small and hand-adjudicated by one researcher; that is the durability mechanism and also its current ceiling.

Reproducibility

sota_bench is open (Apache-2.0): the labeled corpus with pinned commits, the non-LLM scorer, the deterministic CVSS-recompute and triad checkers, the loop, and the pinned 2026-06-03 baseline. Re-run the loop against the next frontier model and the delta updates itself. github.com/matte1782/sota-bench

References

• Anthropic, Coordinated Vulnerability Disclosure dashboard (red.anthropic.com/2026/cvd/): 23,019 candidate findings, a hand-assessed slice (about 1,900 at 90.8% true-positive), 1,596 disclosed vulnerabilities, and the 58.7% / 94.4% severity-agreement figures.
• Google Project Zero, "From Naptime to Big Sleep" (projectzero.google, 2024): autonomous discovery on live targets; benchmark abandonment.
• PrimeVul: a vulnerability-detection benchmark with the VD-S metric and pairwise vulnerable/benign evaluation (arXiv:2403.18624).
• OWASP Benchmark: TP/FP/TN/FN scoring and the Youden index for SAST evaluation (owasp.org/www-project-benchmark).
• CyberGym: real-world AI cyber-capability evaluation at scale; execution-grounded pass criterion (arXiv:2506.02548).
• ZeroPath, "Toward actual SAST benchmarks": positive plus patched-counterpart (exonerated-negative) design.
• FIRST, CVSS v3.1 Specification (first.org/cvss/v3-1/specification-document): the base-score formula and roundup used by the deterministic recompute check.
• "Security in LLM-as-a-Judge" SoK (arXiv:2603.29403): why a validation/calibration layer must not be model-judged.