AI code review false positives: how we measure ours
A false positive is a review comment that is wrong once someone checks it. In our benchmark we sort findings into true, false and could not verify, and for part of the findings no author acted on we also attempted a run; how many got a run differs by comparison and by side. On the 56 pull requests shared with CodeRabbit in our July 20, 2026 readout, Veridical raised 148 findings that matched no change the author made; a run confirmed 68, refuted 19 and could not be set up for 58, and no run was attempted on 3.
True, false, could not verify
- True
- The finding describes a real problem. It counts as true when the author later changed the code the way it suggested, when a build-and-run check reproduced it, or, where no run settled it, when a grader judged it real.
- False
- The finding is wrong once someone checks it: a build-and-run check refuted it, or a grader judged it noise. A comment that is merely unwelcome is not false; a comment that is wrong is.
- Could not verify
- Nobody could settle it: the check could not be set up or did not decide the question. The table below reports these separately; in the credited score they keep the grader's verdict.
In the product, a finding that cannot be supported is withheld rather than posted as a bug.
Why “acted on” is not enough
Many tools report how often developers act on their comments. That number is easy to collect and it misleads in both directions.
- It overstates. Authors change code for comments that are trivial or wrong, and a generous match counts a partial overlap as a hit.
- It understates. Real bugs that ship anyway count as misses, and a comment the author never saw cannot be acted on at all.
So we keep the acted-on view as the strict lens and add a second one, credited-BT, that checks the comments nobody acted on.
How we measure
- Credited-BT
- A comment counts as correct when it matches a change the author made after review, or when a grader judged the issue real. Where the issue could be built and run, the run overrides the grader: confirmed counts, refuted does not. The same rules apply to both sides, but how many findings a run was attempted on differs by comparison and by side, so each comparison prints its own counts.
- Strict
- A comment counts as correct only when it matches a change the author made after review. The tool's own comments were visible to the author and ours were not, so this lens favours the tool whose comments the author saw.
- Readout
readout_20260720_191140.json, generated , protocol 1.6.2, scored byofflinebeat/h2h/score_h2h.py. Precision and recall are averaged per pull request; F1 is computed from those averages. The 95% confidence interval (CI) is a paired bootstrap over pull requests with 3000 resamples. - The study was pre-registered on with 3 comparators (CodeRabbit, cubic, Gemini Code Assist) and later extended to 5 (Greptile, OpenAI Codex code review added). The pre-registered headline was the strict lens. We lead with credited-BT because strict scores comments against changes the author made after seeing the other tool's comments.
- Pull requests: merged since March 1, 2026 in public repositories where the tool posted at least one comment, with 2 to 12 commits, at most 15 changed files and fewer than 800 added lines. Veridical reviewed the same pull requests afterwards, from the state before review.
- Each comparison is still below the protocol's first planned checkpoint of 150 pull requests. These readings come before that checkpoint, so treat them as early.
- The Veridical side was our July 2026 research pipeline. It predates the production engine that reviews pull requests today, so read these as dated measurements, not as a statement about today's reviews.
- The CI covers which pull requests were sampled. It does not cover grader variation: a model grades the findings no author acted on, and grading the same findings again can move credited scores. We have not yet published a replicate-grading measurement, so treat small credited differences with care.
- We claim a difference only when the 95% CI on the named lens excludes zero and at least 30 pull requests are shared. A lead for Veridical is claimed only over CodeRabbit, cubic and Greptile, the tools our claim rule names, and only when its interval starts at least one point above zero; any other lead for Veridical is printed as measured, not claimed. A lead for the other tool is stated whenever its interval excludes zero. Otherwise we print the numbers and claim neither side.
- A difference is Veridical's F1 minus the tool's, computed before rounding, so it can differ in the last digit from the gap between the two printed F1 values. A claimed difference is printed with its interval oriented to the side that scored higher; every other interval carries a plus or minus sign.
Current figures
From our readout of (protocol 1.6.2). Each row is its own set of shared pull requests, so never add rows together. No interval was computed for precision alone, so these tables claim no difference between tools; the F1 results with intervals are on the comparison hub. Credited precision rests on the grader alone wherever no run was attempted, and the second table gives the run counts.
Precision by lens
| Pull requests shared with | Shared PRs | Credited-BT precision, Veridical vs tool | Strict precision, Veridical vs tool |
|---|---|---|---|
| CodeRabbit | 56 | 85.0 vs 84.9 | 33.6 vs 76.2 |
| cubic | 43 | 89.0 vs 84.3 | 35.5 vs 64.9 |
| Gemini Code Assist | 32 | 87.0 vs 83.6 | 40.1 vs 80.5 |
| Greptile | 64 | 86.4 vs 92.4 | 35.5 vs 72.9 |
| OpenAI Codex code review | 8 | 95.8 vs 87.5 | 42.9 vs 75.0 |
Checking the findings nobody acted on
For some of the findings that matched no change the author made, we attempted a run that would confirm or refute the finding. The same rules apply to both sides, but how many findings a run was attempted on differs by comparison and by side, as the last column shows. No run was attempted on Greptile's unmatched findings, so the grader alone scored them. In the OpenAI Codex code review comparison no run was attempted on either side, so the grader alone scored those findings.
| Pull requests shared with | Side | Findings no author acted on | Run confirmed (true) | Run refuted (false) | Run could not be set up | No run attempted |
|---|---|---|---|---|---|---|
| CodeRabbit (56 PRs) | Veridical | 148 | 68 | 19 | 58 | 3 |
| CodeRabbit | 27 | 6 | 0 | 21 | 0 | |
| cubic (43 PRs) | Veridical | 125 | 40 | 8 | 39 | 38 |
| cubic | 24 | 4 | 2 | 10 | 8 | |
| Gemini Code Assist (32 PRs) | Veridical | 80 | 33 | 8 | 26 | 13 |
| Gemini Code Assist | 14 | 1 | 1 | 10 | 2 | |
| Greptile (64 PRs) | Veridical | 172 | 13 | 8 | 17 | 134 |
| Greptile | 27 | 0 | 0 | 0 | 27 | |
| OpenAI Codex code review (8 PRs) | Veridical | 16 | 0 | 0 | 0 | 16 |
| OpenAI Codex code review | 2 | 0 | 0 | 0 | 2 |
Where a run could not be set up, the credited-BT score falls back to the grader's verdict for that finding. Counts are raw: they are not adjusted for how many comments each side made.
Reproduce it on your own repository
- 01 Pick 20 to 50 merged pull requests from your own repository, across the kinds of change your team ships.
- 02 For each one, freeze the base and the first reviewed commit, and list what the author changed after review. Keep that list away from whoever grades.
- 03 Run each tool on the same commits. Collect the comments it posted, not its intermediate notes.
- 04 Match comments to the author's later changes. For the rest, try to reproduce each one with your own build and tests; mark it true, false or could not verify, and keep the command and its output.
- 05 Per pull request, compute precision (correct comments over posted comments) and recall (issues found over issues present), average them across pull requests, and bootstrap over pull requests for an interval.
A shorter version, two tools on one repository for two weeks, is on each comparison page.
Questions
- What is a false positive in AI code review?
- A review comment that turns out to be wrong when someone checks it: the bug it describes does not exist, or the fix it suggests would break something. We separate those from comments nobody could settle, which we label could not verify.
- Is a comment the author ignored a false positive?
- Not necessarily. Authors skip real issues for many reasons, and they act on wrong or trivial comments too. That is why we do not use acted-on rates alone: we also check the comments nobody acted on. A grader judges each one, and where a run confirmed or refuted it, the run decides. How many got a run differs by comparison and by side.
- What is your false positive rate?
- It depends on the lens, so we print both. On pull requests shared with CodeRabbit, Veridical's precision (the share of its findings counted correct) was 85.0 on credited-BT and 33.6 on strict, from our readout of July 20, 2026. On strict, a finding counts as incorrect whenever it matched no change the author made, which does not make it false. The tables on this page give every tool's figures with the same lenses.
- Why are your strict numbers lower than other tools'?
- Strict counts only comments the author acted on. In our study the other tool's comments were posted on the pull request and ours were not, so the author could act on theirs and never saw ours. Strict therefore favours the tool whose comments were visible, which is why we lead with credited-BT and print strict beside it.
- Do these numbers describe today's Veridical reviews?
- No. The Veridical side was our July 2026 research pipeline, which predates the production engine that reviews pull requests today. Each figure carries its readout date for that reason.
- Can I check your numbers?
- You can reproduce the method on your own repository with the five steps on this page. For questions about our protocol or a specific figure, write to contact@veridical.dev.
See what your next pull request's findings rest on.
You sign in with your GitHub account. The first time, GitHub asks you to authorize Veridical.