Field guide · 2026
How to evaluate AI code review tools.
The best AI code review tool is not the one with the longest feature list. It is the one that finds the defects your team cares about, publishes evidence maintainers can trust, and fails visibly when the review is incomplete.
By Łukasz Szafrański · 8 min read
The short answer
Run candidate tools on the same historical pull requests, hide the answers until reviews are sealed, and grade the final comments—not the model's intermediate guesses. Favor defect recall, publication precision, exact-head safety, and concrete evidence over comment volume.
Six criteria that expose the difference
Recall on real defects
Does it find defects maintainers would actually fix?
Demand this proof: A blinded, same-PR corpus with an explicit matching policy and missed-defect accounting.
Precision at publication
How much maintainer attention does each correct finding consume?
Demand this proof: Final posted comments—not raw candidates—graded for correctness and actionability.
Repository context
Can it follow unchanged callers, contracts, configuration, and dependencies?
Demand this proof: A test PR whose defect cannot be proven from the changed hunk alone.
Executable evidence
Can it reproduce a claim with project-native tools when the repository permits it?
Demand this proof: A receipt showing the command, environment, result, and exact reviewed head.
Head safety
Can feedback from an obsolete commit appear current?
Demand this proof: Push a new head during review and inspect how the result is marked or withheld.
Failure disclosure
What happens when source, checks, or model calls are incomplete?
Demand this proof: Coverage and degradation are visible without presenting internal model telemetry as evidence.
Feature claims are not quality evidence
Weak buying signal
- “Understands your whole codebase”
- “Uses multiple agents”
- “Supports every language”
- Large raw comment counts
Strong buying signal
- Same-PR measured precision and recall
- Public, inspectable review receipts
- Exact-head and partial-coverage behavior
- Concrete source or reproducible evidence
A reproducible five-step evaluation
- 01Select 20–50 already-merged pull requests across the languages and change types your team ships.
- 02Freeze the repository base and head, then create one defect/action inventory without showing it to the reviewers.
- 03Run every tool on the same source and configuration. Count failures, empty reviews, and latency—not just comments.
- 04Grade final published findings blind to vendor. Match mechanism, consequence, and affected code region.
- 05Report per-PR precision, recall, F1, security-critical misses, cost, and time to first useful finding.
Avoid the convenient benchmark trap
Offline datasets are useful for iteration, but they can reward leaked labels, synthetic defects, or a matching policy that does not resemble maintainer action. Keep a fresh, sealed cohort and treat live, same-head behavior as a separate release gate.
See our own measurement basis
We publish the distinction between measured head-to-head results, projections, and public-board listings.