Trail of Bits published a post on September 15 arguing that the conclusions in 1Password's AI patch benchmark report don't hold up. 1Password's headline number was a 26% "clean fix rate." Trail of Bits re-ran the scoring on the same dataset and got 2,634 of 3,067 patches successfully blocking their paired exploit — 86%.
The two figures differ by 60 percentage points, and the gap comes down to how the results were scored.
Four problems Trail of Bits flagged
Trail of Bits lists four issues:
- The sample was too narrow. The entire benchmark used only six complex vulnerabilities, with fix rates swinging from 3% to 60% between them. Averaging six wildly different results into a single "AI patching capability" score lets sampling noise drown out the signal.
- Some prompts were flawed by design. 22% of the data came from trials where the prompt deliberately steered the agent toward the wrong fix. Those trials measure whether a model can be misled, not whether it can fix the bug correctly.
- Tooling was stripped out. In 36% of trials, the model wasn't allowed to compile or test its own code. Human developers fixing the same vulnerabilities under those constraints wouldn't score much better.
- Configurations weren't consistent. Different models ran under different reasoning settings, which makes the scores not directly comparable across models.
The post is credited to four authors: Anish Naik, Dan Guido, Benjamin Samuels, and Marcelo Morales. 1Password has not publicly responded to the re-analysis so far.
The human baseline
The more persuasive part of the post is the control group it supplies. Trail of Bits cites data showing that developers patching a security vulnerability under ideal conditions fail on their first attempt about 12.5% of the time — roughly one miss in eight.
That baseline is what gives "AI can't patch cleanly" any real meaning. A 26% clean-fix rate looks like a failing grade on its own. Placed next to an 87.5% human baseline, an 86% AI score tells a different story: roughly on par, not yet ahead.
Trail of Bits also shared numbers from its own Patch the Planet project: 186 pull requests submitted, 126 merged upstream, a 67.7% merge rate; of the merged patches, 72.2% needed no further security revisions. That's a different yardstick from the benchmark trials — it measures whether real maintainers accept a patch, which is closer to production reality than merely blocking an exploit.
Two skills released alongside the post
The post also announced two agent skills: post-patch-validation, which has an agent test and verify its own work before submitting a patch, and review-walkthrough, which walks engineers through a code change in logical order.
Releasing those two skills is itself a response to the benchmark. In 36% of the disputed trials, the model was barred from compiling or testing code — and the first new skill is built specifically around pre-submission validation. The implication: patch quality depends heavily on whether the agent gets to check its own work.
What to look for in benchmarks like this
AI vulnerability-patching scorecards have multiplied this year, and their methodologies vary widely. This post effectively hands readers a checklist: how many samples were used, whether any prompts were designed to mislead, whether models were allowed to compile and test, whether reasoning configurations were held consistent across models, and whether there's a human baseline for comparison. Miss any one of those five, and a published percentage isn't enough on its own to base a tooling decision on.
More practically, both skills are public releases that can be wired into an existing code review pipeline without waiting on anyone's approval. How much benefit that brings depends heavily on the codebase in question, and there's no ready-made answer for that yet.
Sources: Trail of Bits official blog, CocoLoop, 1Password benchmark report; the four methodological objections, the 3,067-vs-2,634 patch count, and the 186/126 Patch the Planet merge figures all follow the original blog post.