Measured against
everyone else, in public
Ozone against nine other AI code reviewers on 165 reviews of real CVEs. Every harness and every finding is published.
Benchmark Date: August 2026
Ozone on the OpenSSF CVE Benchmark
We took 82 real vulnerabilities from the Linux Foundation's OpenSSF dataset and showed each one to ten AI code reviewers, as an ordinary pull request. Then we showed them the patched version too, to see who cries wolf. Here is how many each one caught.
Ozone ranks #1 of 10 AI code reviewers.
It found 63 of 82 real vulnerabilities with 23 false alarms.
The ranking is by F1 score, the standard measure that rewards both catching vulnerabilities and not crying wolf. Ozone scores 75.0%. Cursor Bugbot caught three more but raised 42 false alarms, so it ranks third. Claude Code found 35. CodeRabbit found 16.
| # | Tool | F1 score | Vulnerabilities caught | False alarms | |
|---|---|---|---|---|---|
| 1 | Ozone | 75.0% | 63 of 82 | 23 | |
| 2 | DeepSource | 70.6% | 48 of 82 | 6 | |
| 3 | Cursor Bugbot | 69.5% | 66 of 82 | 42 | |
| 4 | Devin | 66.7% | 47 of 82 | 12 | |
| 5 | Codex | 63.9% | 46 of 82 | 16 | |
| 6 | Claude Code | 55.1% | 35 of 82 | 10 | |
| 7 | Greptile | 50.4% | 35 of 82 | 22 | |
| 8 | GitLab Duo | 46.2% | 30 of 85 | 15 | |
| 9 | CodeRabbit | 29.4% | 16 of 82 | 11 | |
| 10 | Semgrep CE | 24.0% | 12 of 82 | 6 |
Competitors' findings are their own published output, released by DeepSource in April 2026. Only the grading is ours, applied identically to all ten tools and blind to which tool produced a finding. F1 is the harmonic mean of recall (the share of vulnerabilities caught) and precision (the share of flags that were real). Ozone: 76.8% recall, 73.3% precision.
Missing a vulnerability ships a bug
A reviewer that catches 16 of 82 is quiet, and quiet looks clean right up until something reaches production. This is the column most tools would rather you did not read.
A false alarm costs you trust
Every wrong flag is an engineer investigating nothing. Enough of them and the team stops reading the tool, at which point its detection rate stops mattering entirely.
A third of this benchmark cannot measure false alarms.
The fixture swaps only the one file each CVE record names. For 27 of the 85 vulnerabilities, the real fix changed other source files too — so the "patched" version is still exploitable, and a reviewer that correctly says so is counted as raising a false alarm. It penalises whichever tool detects the most.
We also found that telling the grader which version it is inspecting hands it the answer: our first attempt returned zero false alarms for all nine tools, which is impossible. And DeepSource's published summary reports a perfect score for CodeRabbit where their own raw data shows four false alarms.
None of that changes the order of this table, and the numbers above are the uncorrected ones. We raise it because a benchmark nobody audits is a benchmark nobody should trust — including ours.
What this does not show
- JavaScript and TypeScript only. It says nothing about our coverage of Python, Go, Java, Rust or Solidity.
- The CVEs are public and date from 2016 to 2021, so models have likely seen them in training. That inflates all ten tools equally — the ranking holds, the absolute figures are not real-world detection rates.
- DeepSource ran CodeRabbit and Claude Code through a CLI rather than as pull request reviews. Both are pull-request products, so their figures here are likely understated and our largest margins are the least meaningful ones in the table.
- Competitor runs are from April 2026; ours are from August 2026. These are not contemporaneous builds.
- The grader is GPT-5.6, the same model family Ozone reviews with. It never sees code or a tool's name, and it agrees with DeepSource's Claude grader on 88.6% of rows, but a cross-family grading would be stronger. The harness accepts a Claude grader so anyone can produce one.
- DeepSource's own chart shows higher scores for every tool. Its grader was told which files were already patched, which suppresses false alarms; ours is not. Both gradings are in the repository, and the ranking holds under either.
See the findings on your own code.
A 2017 npm package is a fair test. Your repository is the one that matters.
Get started free