Measured against
everyone else, in public
Ozone against nine other AI code reviewers on 165 reviews of real CVEs. Every harness and every finding is published.
Benchmark Date: August 2026
Ozone on the OpenSSF CVE Benchmark
We took 82 real vulnerabilities from the Linux Foundation's OpenSSF dataset and showed each one to ten AI code reviewers, as an ordinary pull request. Then we showed them the patched version too, to see who cries wolf. Here is how many each one caught.
Ozone found 63 of 82 real vulnerabilities.
Claude Code found 35. CodeRabbit found 16.
One tool caught more — Cursor Bugbot, at 66 — but it raised 42 false alarms against Ozone's 23. Every other reviewer tested missed at least a third more vulnerabilities than Ozone did.
| Tool | Vulnerabilities caught | False alarms | |
|---|---|---|---|
| Cursor Bugbot | 66 of 82 | 42 | |
| Ozone | 63 of 82 | 23 | |
| DeepSource | 48 of 82 | 6 | |
| Devin | 47 of 82 | 12 | |
| Codex | 46 of 82 | 16 | |
| Claude Code | 35 of 82 | 10 | |
| Greptile | 35 of 82 | 22 | |
| GitLab Duo | 30 of 85 | 15 | |
| CodeRabbit | 16 of 82 | 11 | |
| Semgrep CE | 12 of 82 | 6 |
Competitors' findings are their own published output, released by DeepSource in April 2026. Only the grading is ours, applied identically to all ten tools and blind to which tool produced a finding. In percentages, Ozone is 76.8% recall and 73.3% precision, the highest F1 of the ten at 75.0%.
Missing a vulnerability ships a bug
A reviewer that catches 16 of 82 is quiet, and quiet looks clean right up until something reaches production. This is the column most tools would rather you did not read.
A false alarm costs you trust
Every wrong flag is an engineer investigating nothing. Enough of them and the team stops reading the tool, at which point its detection rate stops mattering entirely.
A third of this benchmark cannot measure false alarms.
The fixture swaps only the one file each CVE record names. For 27 of the 85 vulnerabilities, the real fix changed other source files too — so the "patched" version is still exploitable, and a reviewer that correctly says so is counted as raising a false alarm. It penalises whichever tool detects the most.
We also found that telling the grader which version it is inspecting hands it the answer: our first attempt returned zero false alarms for all nine tools, which is impossible. And DeepSource's published summary reports a perfect score for CodeRabbit where their own raw data shows four false alarms.
None of that changes the order of this table, and the numbers above are the uncorrected ones. We raise it because a benchmark nobody audits is a benchmark nobody should trust — including ours.
What this does not show
- JavaScript and TypeScript only. It says nothing about our coverage of Python, Go, Java, Rust or Solidity.
- The CVEs are public and date from 2016 to 2021, so models have likely seen them in training. That inflates all ten tools equally — the ranking holds, the absolute figures are not real-world detection rates.
- DeepSource ran CodeRabbit and Claude Code through a CLI rather than as pull request reviews. Both are pull-request products, so their figures here are likely understated and our largest margins are the least meaningful ones in the table.
- Competitor runs are from April 2026; ours are from August 2026. These are not contemporaneous builds.
See the findings on your own code.
A 2017 npm package is a fair test. Your repository is the one that matters.
Get started free