Measured comparison · August 2026
Semgrep vs AI code review, on the same 82 vulnerabilities
Semgrep's community ruleset caught 12 of 82 real vulnerabilities, with 6 false alarms — the joint-cleanest result in the test. Ozone caught 63 with 23 false alarms.
This is not a like-for-like fight, and framing it as one would be dishonest. Semgrep is deterministic pattern matching: it finds precisely what its rules describe, every time, for free, in milliseconds. That is a different product from an agent that reasons about a codebase.
The useful question is not which wins. It is what a rule can express, and what falls outside it.
| Tool | Vulnerabilities caught | False alarms | |
|---|---|---|---|
| Semgrep CE | 12 of 82 | 6 | |
| Ozone | 63 of 82 | 23 | |
| Cursor Bugbot | 66 of 82 | 42 | |
| DeepSource | 48 of 82 | 6 |
OpenSSF CVE Benchmark, 82 real JavaScript and TypeScript vulnerabilities, each also shown patched to count false alarms. Competitors' findings are their own published output (DeepSource, April 2026); the grading is ours, applied identically to all ten and blind to which tool produced a finding.
What rules are unbeatable at
Deterministic, auditable, fast, and free. A Semgrep rule gives the same answer on every run, can be read and reviewed by a human, lives in version control, and executes in CI in seconds. No AI reviewer in this test can claim any of those.
For known-shape problems — a banned function, a hardcoded secret, a dangerous API called without its guard — a rule is simply the right tool, and 6 false alarms across 82 patched files shows how precise that can be.
What falls outside a rule
The 70 vulnerabilities Semgrep missed are mostly not pattern-shaped. Whether a path traversal is exploitable depends on what sanitisation ran three functions earlier. Whether a prototype pollution matters depends on whether attacker input reaches the merge.
Those questions are about reachability across files, and a rule that tried to express them would either miss the variants or fire on everything. This is the gap AI reviewers are aimed at — and the 63-versus-12 difference is roughly the size of that gap.
The sensible answer is both
Rules for the known and the mechanical, running on every commit at no cost. A reasoning reviewer for the things a rule cannot express, on pull requests where the expense is justified.
Anyone selling you the idea that AI review replaces static analysis is overselling. The measured result here is that they fail differently, which is exactly why they compose well.
When to choose Semgrep
- You need deterministic results you can audit, version and explain to a compliance reviewer. No AI tool here offers that.
- You want zero marginal cost per scan. Semgrep CE is free and fast enough to run on every commit.
- Your priority is enforcing known internal standards rather than discovering unknown vulnerabilities — the case rules were built for.
- You are unwilling to accept a false alarm rate above single digits. Semgrep tied for the cleanest in the test.
Common questions
- Does AI code review replace Semgrep?
- No. Semgrep caught 12 of 82 vulnerabilities with 6 false alarms, and the ones it caught it will catch deterministically every time for free. AI reviewers caught up to 66 but cost money per review and vary between runs. They fail differently and work well together.
- Why does Semgrep miss so many vulnerabilities?
- Most of the vulnerabilities in this benchmark depend on reachability across several files — whether attacker-controlled input can arrive at a dangerous sink. That is difficult to express as a pattern without either missing variants or matching everything.
Check the numbers yourself.
The harness, the fixtures, every raw finding and every grader verdict are published. Re-run it and tell us if you get something different.