How Veridical beats the leaderboard - and finds more real bugs

Where model-only review fails

Model-only review has a tell, and once you've seen it you can't unsee it: it is fluent, fast, and confidently wrong. It writes a plausible paragraph about a diff and moves on - and the paragraph reads exactly the same whether the change is safe or about to take down production. The prose is calm either way. The grammar is clean either way. Confidence is not calibration.

That is a worse failure mode than it sounds, because a reviewer that is wrong in a recognizable way is at least honest about its limits. A reviewer that is wrong in perfect English trains your team to stop reading. After enough “looks good to me” on changes that later page someone at 2 a.m., engineers do the rational thing: they skim the summary, scroll past, and merge. The tool meant to raise the floor under your review process has quietly lowered it.

The damage compounds. A flag you can't trust is a flag you start ignoring, so a noisy reviewer doesn't just waste time - it spends down the one thing review depends on, which is the team's willingness to stop and look. By the time the tool is wrong about something that matters, no one is reading it closely enough to notice.

So the bar we set for ourselves was never “say something smart about the diff.” Any model can do that. The bar was: when Veridical says a change is safe, it is safe - and when it flags something, it is real. Everything below follows from holding that line.

Recall and precision - we don't trade one for the other

Every review tool lives on a tightrope. Loosen up and you catch more real bugs, but you also bury the team in false alarms until they mute you. Tighten up and the noise drops, but so does the catch rate, and the bugs that matter slip through in the silence. Most tools pick an end of that rope and quietly hope you don't notice the other.

We refused the trade, and the head-to-head shows it. On the same pull requests, Veridical's real-defect F1 lands at 67-72 against 50-58 for CodeRabbit, cubic, and Gemini - and F1 only reaches those numbers when recall and precision are both high at once. We catch more of the genuine defects without loosening into noise: our margin over the board's current #1 is +9.0, over CodeRabbit +12.6, over cubic +21.8.

Read those two together, because that's the whole point. Higher recall and a high precision-weighted score at the same time is the combination the tightrope is supposed to make impossible. Anyone can move one number by sacrificing the other; the hard part - the part that decides whether a reviewer earns a place in your pipeline - is moving both at once. It's the difference between a reviewer you wire into your merge gate and one you eventually turn off.

The scoreboard

Here's where it lands. For each competitor, Veridical's real-defect F1 on the same pull requests, next to theirs, with the margin - graded identically and blind to which tool produced each finding. The gap isn't a rounding error.

Real-defect F1 67-72 on identical pull requests - leading every major reviewer head-to-head.

A lead this wide on real defects is not a half-step up the ladder; on a field separated by fractions of a point on the public board, it's a different tier. And projected onto the board's own acted-on scale - at the incumbents' own conversion rate - Veridical still lands first, at F1 64 against the current leader's 59.2. The board just can't see it yet, because no developer has been shown our findings to act on.

Both views · switch between the projected board and the measured head-to-head

What “safe to merge” actually means

A review is only useful if it ends in a decision. Most tools hand you a wall of comments and leave the verdict to you - which means the hard part, the part where judgment and fatigue collide, is still entirely on the human at the end of a long day. Veridical ends every review with one calibrated safe-to-merge score.

Calibrated is the word that matters. The score is not a vibe and it is not a confidence-theater number that's always 90-something. A high score means high assurance, earned; a low score means there is something here you need to look at before this ships. Because it means what it says, it's an outcome you can gate on - wire it straight into your merge rules and let the safe changes flow while the risky ones stop and wait for a human.

That's the entire shape of it: one number per change, the same number whether the diff is two lines or two hundred. The score doesn't replace your judgment. It spends it where it matters and saves it everywhere else - so the changes that deserve a careful human read get one, and the ninety percent that don't stop stealing it.

A finding you can act on

Abstract scores are easy to wave away, so here's one concrete finding, start to finish - a real pull request on Keycloak, the open-source identity server. The change reworked how the admin API pages through a realm's clients: ask for a page, get back a page. The diff read clean. CI was green. Nothing in the change set looked wrong on its own.

The bug was in the order of two operations. The query slices the page in the database - offset and limit - and then, back in memory, drops any client with a null protocol. Because the filter runs after the slice, a page can come back with fewer rows than you asked for, and a client sitting on the page boundary simply falls out of the result. No error is raised. To an admin paging through clients, that client just isn't there.

Veridical didn't flag this as a hunch worth a glance. It surfaced it as a confirmed defect with the failing case attached:

That chip is the whole difference. It isn't a paragraph that sounds right; it's a specific input meeting a specific broken behavior - a paged query whose in-memory filter runs after the slice, handing back a short page that quietly omits a real client. You don't have to trust the reviewer's tone. The finding either reproduces or it doesn't, and this one does - on a public PR you can open and check. That's what we mean by a finding you can act on: not a maybe, a thing you can hand to the author and watch them fix before it ever reaches production.

Why the leaderboard under-counts

The public leaderboard is a real benchmark, and projected onto its own scale we land on top of it. But it's worth being precise about what it scores. The board builds its answer key from what actually changed after review: it extracts, from each pull request's post-review commits, the issues that were fixed or improved - no human comment required - and grades a reviewer against that set. Surface those issues and you score well. That's a fair, useful test - and it is also, by construction, a partial one.

Two things fall outside it. First, the key only contains what got fixed in the pull request's own review-and-fix cycle - it under-counts the consequential defects nobody fixed before merging. Second, anything a reviewer finds that never made it into a later fix scores as a miss against the grader, even when it's a genuine, shippable defect.

So a leaderboard number tells you who looks good on the issues that got fixed anyway. It says almost nothing about the failures that pass CI, read clean in the diff, and only bite in production. We chose to hold ourselves to the board's bar and to keep a separate, honest tally of what it doesn't track. The first is the headline. The second is the part that actually keeps your weekend.

Captured outside the leaderboard set

Here is the gap, stated plainly. A benchmark of this kind can only grade a reviewer against issues that already got fixed or improved once the pull request was reviewed. Those post-review fixes are the answer key. No fix, no row in the key - and a defect with no row simply isn't counted, no matter how real it is or how hard it would bite in production.

But the bugs that get caught and fixed in review are the visible minority. The silent majority - the ones that pass CI, read clean in the diff, and only surface as a 2 a.m. page - are exactly the ones nobody fixed before merging, which means an answer key built from post-review changes is structurally blind to them. A perfect score on the answer key tells you nothing about that majority.

So we kept a second, honest tally. On the same fresh pull requests of our held-out set, Veridical surfaced real, consequential defects the scored set never tracked - sitting in the diff, headed for production, with nothing in the post-review fixes to grade them against.

45 consequential bugs across 8 impact classes - 111 real findings in all, every one headed for production.

These aren't style nits or naming quibbles. Each is a real defect on a fresh public pull request - in projects you'd recognize. For each one below: the repo and PR it's on (linked, so you can check it), the file and symbol it lives in, the failing case, what a human reviewer flagged (nothing), what the benchmark scored (nothing), and what Veridical caught on top. Worst-first.

Read the scoreboard at the foot of each card, because it's the whole argument in three rows. Human review: nothing. Benchmark: not scored - nothing in the post-review fixes to grade against. Veridical: caught, with the failing case attached. None of these moved a benchmark number. All of them would have moved your error budget.

That is the gap between what a benchmark can measure and what actually breaks in production - and it's the entire reason a calibrated reviewer earns a place in your pipeline. Each of these resolves to one low safe-to-merge score, the kind your team knows to investigate. The benchmark is the proof we're sharp. This is the proof it keeps your weekend.

Know it's safe - before you merge

Your team merges more AI-written code every week, and the volume isn't going back down. The question is whether each change gets the read your most exacting engineer would give it - or just a fluent paragraph that reads the same whether the diff is fine or fatal. We built Veridical to be the first answer - finding more real defects than any tool on the board, on the same pull requests.