Notebook · 2026-07-29

I scored my own bug-hunting protocol. It would have killed three of the five real findings.

Two of the three were mine. A self-audit of a verification gate, the two specific ways it failed, and the discipline problem sitting underneath it.

Matteo Panzeri · AI-for-security research

Written with AI assistance. The adversarial reviews referenced throughout were independent adversarial LLM passes, not third-party review. Disclosure: one of the backtest cases is Pillar Security’s work; their researcher Ariel Fogel contacted me after publishing it, credits me in one of their repositories for an unrelated finding of mine, and has invited me to be interviewed. I used their case because it is the clearest public example of the failure mode, and because their own account of what happened in triage is the external evidence for it.

Two hunts in a row returned nothing, and a third produced two candidates that also died. A hardware wallet, a coding agent, an agent SDK. Each write-up concluded, with some confidence, that the surface was well defended.

That is a suspicious kind of result, so I tested the instrument instead of the target. I took the verification protocol I had built, assembled ten cases whose outcomes I already knew, and had each judged against the rules.

It scored 40 percent recall and 50 percent precision. Of the five findings that were genuinely accepted in the real world, the protocol would have killed three. Two of those three were mine.

What the protocol was

After the second empty hunt I wrote a gate: a qualification checklist, ten kill criteria, a mandatory pre-check before any deep analysis, and a dynamic phase with six controls against confirmation bias. Default verdict on any candidate: not a bug. Every candidate had to survive independent reviewers whose job was to refute it.

The goal was to stop me believing my own hypotheses. It is the kind of thing people publish as a methodology post, which is roughly what I was planning to do with it.

The backtest, and what is wrong with it

Ten cases: five accepted, credited or paid; five rejected. Each was judged by a separate analyst who was given the rules and the case and told to apply them mechanically.

The three false negatives were Pillar Security’s Codex git show finding, and two of my own published CVEs.

The design flaw, stated before the result is used for anything. The analysts were blind to each other, not to the outcome. Each was told what really happened and instructed not to let it bias the grading. That is weaker than a blind trial and it biases in a knowable direction: an analyst who knows a case was accepted has a reason to find the protocol at fault, which inflates the measured false-negative rate. I do not know by how much.

n is ten. With five positives, the difference between 40 and 60 percent recall is one case. There are no confidence intervals here and I am not going to manufacture any.

Nine of the ten cases are my own work. Only Pillar’s is external. This is a self-audit, not a study.

The composition is worse than the headline. Both true positives passed only under a charitable reading of a rule that had a defensible kill available. The three true negatives are worth about one clean kill between them: one analyst called their own case close to tautological, another rated their kill 60 to 70 percent reliable, a third was contradicted by what the record actually showed.

So the honest summary is not a rate. It is: on a small, outcome-aware, self-referential sample, the gate killed more real findings than it caught. That is enough to make me stop using it. It is not enough to make it a study.

The two failure modes worth naming

I am deliberately not claiming a principle here. “Controls in one direction move the decision threshold” is the precision and recall tradeoff with a costume on, and I do not need ten cases to rediscover it. What I think is worth writing down is narrower: two specific ways the controls failed, one of which has external corroboration.

Refutation by retry is unfalsifiable in reverse

One control, D6, said: re-run adversarially with the goal of making the finding fail.

Against a deterministic bug that is a good test. Against a probabilistic one it is not a test at all. If a finding depends on a model complying with an injected instruction, and the model refuses some fraction of the time, then “try to make it fail” always succeeds. You rerun until a refusal and record a refutation. D6 can refute any model-mediated finding, including every true one.

The external evidence is not mine. When Pillar reported the git show bug, the triage team could not reproduce it, because the model refused the injected instruction on their attempt. Pillar’s account makes the point directly: model refusal is probabilistic, and running the prompt enough times gets compliance. The finding was assessed at CVSS 8.6 and fixed in v0.95.0.

So a human triage process recovered from the probabilistic case, and the automated one I built was constructed so that it could not.

The fix is not to delete the control. It is that a probabilistic effect has a rate, and a rate is a measurement, not a refutation. One failed run means nothing.

Vendor text describing a mechanism is not vendor acceptance of a consequence

The second rule, K3, killed any candidate where the vendor had documented the behaviour as intentional.

That conflates two different statements. “This mechanism is deliberate” is a design decision and a fair kill. “These commands cannot do harm regardless of their arguments” is a safety claim, and a safety claim is not a stop sign, it is the target. The vendor removed a guard because they believed it. If the belief is false, the documentation is the aggravating factor, not the reason to walk away.

Pillar’s finding is exactly that shape. The tool ships an allowlist of commands that skip both the sandbox and the approval prompt, on the stated theory that they cannot do harm whatever the arguments. git show is on it because it displays a commit. But git show --output writes to an arbitrary file and --format controls the content, so it can plant a malicious external diff tool that fires later when the developer runs git diff. Their line for it is better than mine: the allowlist trusted a name, and the risk was always in the invocation.

My rule would have read the vendor’s documentation of that allowlist and stopped.

The scoreboard for the part I was proudest of

The six anti-bias controls were the newest and most carefully designed part of the protocol. Across the backtest they caught 0 of the 4 rejections and killed 1 of the 1 true positive they touched. Net contribution: negative.

Why none of this announces itself

A false positive is visible. You send a weak report, someone tells you no, you lose a little credibility and you learn something.

A false negative is invisible. The candidate dies in your own notes.

Which means an over-aggressive gate does not look like a broken gate. It produces zeros that look like honest results. My hunts returned nothing and each memo concluded the surface was saturated. Without an external check I had no way to tell “there was nothing there” from “I killed everything”.

The thing the gate could not measure at all

There is a deeper problem that the correction does not touch, and it took a separate audit of my own submission history to see it.

Every instrument in the protocol measures whether a bug is true. Programs pay for findings that are true and novel and above the vendor’s bar. Novelty I could only approximate through public visibility, and impact tier I was not measuring at all.

Worse, the gate is mildly anti-correlated with getting paid. One of my rejected cases scored a perfect pass on every qualification check: attacker-controlled input at the entry point, readable open-source code, exploitable in stock defaults, reproducible in ten minutes. Those are exactly the properties that make a bug easy for every other researcher to find first. It was closed as a duplicate.

A gate that selects for verifiability selects for findability. I had not noticed that they are the same axis.

Testing whether the correction was real

Loosening rules always produces more findings, so “I fixed my methodology and now it works” is worth nothing on its own.

I re-ran the corrected rules over all eight candidates my hunts had killed, with an explicit instruction that confirming a correct kill was a valid and useful outcome.

Five stayed dead. Three reopened.

The five that stayed dead got stronger, because the corrected rules forced someone to read layers the original kills had only assumed: in one case a rendering layer nobody had opened, in another an on-chain program the kill had asserted would reject the input. Both confirmed the kill.

All three reopenings traced to K3, the mechanism-versus-consequence rule.

How much should you believe that? Less than I would like. Three events is not a distribution, and a rule-specific change producing rule-specific reopenings is close to tautological: I changed K3, so of course the cases K3 had killed are the ones that moved. What it rules out is the crudest failure, that I relaxed everything and took the yield as vindication. It does not show the new rules are right.

And the rule I was proudest of did nothing. The symmetric-burden rule, the one requiring a kill to carry the same evidentiary weight as a finding, changed the confidence attached to eight kills and the verdict of none of them. It is a good rule. It is not why anything reopened.

The part where I stopped following my own method

I took the strongest reopened candidate and verified it.

To be precise, because the distinction is the whole point of what follows: I executed the layer where the defect lives. Four tests, all passing, confirming the behaviour I had claimed. The rest of the chain, from that behaviour to the impact, I traced by reading the source and citing lines. I did not run the end-to-end attack.

The tests also disproved one of my own assumptions, which cut both ways. I had claimed the attack required a specific precondition, and it did not, which made the finding easier to trigger. It also removed the argument I had been using to distinguish my case from a related public report, which made the duplicate risk worse. I recorded both.

Then my red team reviewed the report and said, in substance: do not send this until you have run it end to end, because the program’s own scope grants credit for demonstrating the impact, and you are demonstrating a mechanism rather than an impact. It named the outcome precisely.

I sent it anyway. There was an open vendor pull request touching the same code and I did not want to be duplicated.

It was closed without award, on the grounds the review had predicted almost word for word.

I am not going to describe the finding or quote the correspondence, because submissions to that program are confidential under its terms and that applies to closed submissions too. The relevant part is not the bug. It is that the reasoning for rejecting it existed, in writing, in my own files, before I clicked send.

The third time

Afterwards I went looking in my own records, and this was not new.

• A submission where the ledger entry written before sending predicts it will be closed as out of scope. It was closed as out of scope.
• A hardware wallet finding where the pre-send memo says the vendor will probably decline it and warns me not to oversell the impact. The vendor declined it, giving essentially the reason the memo anticipated.
• This one, where an automated review named the rejection in advance.

Three times, three different mechanisms: an ad hoc note, a structured field in a ledger, and an automated review panel. The prediction machinery improved at every step. The behaviour did not change once.

That is the finding I did not want. I had been treating this as an information problem and building better instruments for it. The binding constraint was never information. It was acting on information I already had. A better gate does not fix that, and building one is a comfortable way of not addressing it.

What I would tell someone building the same thing

• Score your gate before you trust it. A methodology you have not measured against known outcomes is a preference.
• Count your controls in both directions. If every one makes it harder to believe and none makes it harder to dismiss, you have built a machine whose output is zero.
• Any control of the form “retry until it fails” is unfalsifiable in reverse against anything probabilistic. Measure the rate.
• A vendor’s safety claim is a target. A vendor’s design decision is a stop sign. They read almost identically and mean opposite things.
• Check whether your gate is selecting for the same property that makes a bug easy for others to find first.
• Give the number that flatters you the same scepticism as the one that does not. I failed this in the first draft of this article and it was pointed out to me.
• Watch the send gate, not only the kill gate. Mine was the one that was actually broken.

Limits

• One researcher’s protocol, scored on ten cases, nine of them his own work, graded by analysts who knew the outcomes.
• The two failure modes are general in shape and one has external corroboration, but nothing here shows how common they are elsewhere.
• The re-adjudication is three events. The corrected protocol has produced exactly one candidate so far and it was rejected.

I have written down the next test before running it: the same method, unchanged, against a target selected mechanically to match the conditions where it has previously worked, with local execution available from the start, and with the conditions under which I would conclude the method does not work fixed in advance. I will publish that outcome either way.