Veridical.dev
Veridical12 min readArticle

Diff-Based Analysis: What It Catches and What It Misses

← All articles

Diff-Based Analysis: What It Catches and What It Misses

Geometric diff hunks opening into repository context

Diff-based analysis examines only the lines a change adds, removes or modifies, and judges the change from that slice alone. It is fast and focused, and it is the right starting point for pull request review. It becomes dangerous when it is also the ending point, because many serious defects live in the code the diff does not show: the callers, interfaces and dependencies that give a changed line its meaning.

This guide explains how a diff is produced, what diff-based analysis does well, where it fails, and how to close the gap without giving up the speed that makes it attractive.

Veridical
Review Pull Requests With Evidence
Veridical reviews GitHub pull requests with verified findings and inline evidence, helping teams catch critical defects before merging.

TL;DR

  • A diff is a text comparison. Diff-based analysis inherits that limit: it sees what changed, not what the change affects.
  • Scoping scans to changed lines is a real speedup. GitHub reports up to 10x shorter scan times for diff-informed CodeQL analysis.
  • Small patches to auth, secrets or shared interfaces need the diff treated as an entry point, then traced through call chains and dependencies.
  • The strongest setup pairs diff-scoped speed with repository context and findings tied to evidence, ending in a score you can gate merges on.

Table of Contents

What diff-based analysis actually is

A diff is the output of a comparison between two versions of a file or tree. Every diff-based tool, from a plain terminal command to an AI reviewer, starts from that same artifact: a list of hunks, each with removed lines, added lines and a few lines of unchanged surroundings.

The comparison itself is not neutral. Git offers several algorithms through the --diff-algorithm option, and Myers, a basic greedy algorithm, is the default. The alternatives are minimal, patience (which anchors on unique lines) and histogram (which extends patience to handle low-occurrence common lines). They can produce different hunks for the same change, which matters when a tool reasons about “what moved” versus “what was rewritten”.

Four Git diff algorithms and their matching approaches

A quick way to see this on your own repository:

git diff --diff-algorithm=myers main...feature > myers.diff
git diff --diff-algorithm=histogram main...feature > histogram.diff
diff myers.diff histogram.diff

On a refactor with repeated braces or boilerplate, the two outputs often split hunks differently. Neither is wrong. Both describe the same change, but a reviewer or a model may read them differently, so it helps to know which one your tooling consumes.

Why teams scope analysis to the diff

The appeal is practical, and it holds up.

Speed. Full scans on every pull request are slow in large codebases. GitHub’s documentation on incremental analysis describes diff-informed analysis, which restricts a CodeQL scan to the lines added or changed in a PR, and cites scan-time reductions of up to 10x compared with analyzing everything. That is the difference between feedback in minutes and a check developers learn to ignore.

Focus. Reviewers have limited attention. Research on vulnerability detection during code review found that simply asking reviewers to focus on security raised the probability of detection eight times, and that the mere presence of a security checklist did not significantly improve the outcome. The lesson for tooling is the same: a narrow, explicit question over a bounded set of changes beats a broad, unfocused one.

Relevance. A finding on a line the author just wrote is a finding the author can act on. A finding in code untouched for three years is, to that author, noise.

Where a diff alone fails

The weakness is structural. Researchers describe a gap between syntactic diffing, which reports text changes, and semantic awareness of behavioral impact. Developers miss security issues when their tools cannot trace how a changed line interacts with the logic around it.

Three patterns show up repeatedly.

The one-line change with a long reach

Consider a change to a helper:

- def normalize_user_id(raw):
-     return raw.strip().lower()
+ def normalize_user_id(raw):
+     return raw.strip()

The diff is two lines. It looks like a harmless cleanup. But if a dozen callers rely on lowercase IDs for cache keys or authorization lookups, this change silently splits one user into several identities. Nothing in the diff says so. Only the callers do.

The safe-looking patch to a sensitive area

Guidance on diff-based versus full-codebase review makes the point directly: when a small patch touches something with outsized consequences, such as authentication or secret management, treat the diff as an entry point rather than the whole picture. Teams that do this well cross-reference the change with call chains and dependency boundaries.

The interface that changed under its consumers

When a function signature, a response schema or a shared type changes, the defect is usually not in the changed file. It is in a consumer that the diff never mentions. This is especially common across service boundaries, which is why we wrote up a microservice code review workflow that catches defects before merge.

Syntactic diff versus semantic diff

The gap has a formal name in program analysis. Microsoft Research work on static analysis describes semantic diff, comparing behavioral summaries rather than lines, as a stronger way to decide whether a change actually alters program behavior, at the cost of more computation than a line-by-line comparison.

The two approaches answer different questions:

Question Line-based diff Semantic or context-aware analysis
What text changed? Exact Exact
Did behavior change? Not answered Estimated from summaries, tests or symbolic execution
Who calls the changed code? Not shown Traced
Does a changed interface break a consumer? Only if the consumer is in the diff Checked across the repository
Cost Very low Higher

The practical conclusion is not to abandon diffs. It is to use the diff to decide where to look, then spend the heavier analysis only on the places that matter.

Making a diff easier to review

Even before any automated analysis, how a diff is presented changes how well people read it.

  1. Order files by dependency, not by name. Documentation for a diff-analysis tool recommends putting new foundations first and callers after them, instead of alphabetical order, which lowers the reader’s cognitive load.
  2. Separate the noise. Generated code and environment-dependent configuration clutter standard diff views. Practitioners should favor tools that filter out non-functional changes or group changes by type, such as new code versus refactor, so attention goes to risk.
  3. Keep pull requests small and single-purpose. A refactor mixed with a behavior change hides the second inside the first.
  4. State the question. Tell reviewers what to look for, such as authorization, input handling or secrets. The research above suggests this beats a generic checklist.

For teams adding AI to this stage, AI-assisted review of code changes raises its own provenance and data-handling questions worth settling before rollout, especially for private repositories.

The context problem in AI review

AI reviewers inherit the diff’s blind spot and add one of their own: how much surrounding material to feed the model. Recent evaluations of AI code auditors report that detection rates for malicious changes fell from 50-60% to 16-22% when review moved from focused per-PR checks to whole-window reviews. More context is not automatically better. Context that is not structured well dilutes the signal, and the same research notes that history-based review can be evaded.

So the goal is not “read everything”. It is to read the right neighbors of each change: the callers of a modified function, the implementations behind a modified interface, the dependencies a changed import brings in. We cover how to judge whether a tool does this in Context-Aware Code Review: What It Takes and How to Verify It.

A practical workflow: diff first, context second

This is the order we recommend for any pull request that touches code you would be uncomfortable shipping wrong.

  1. Scope the fast checks to the diff. Run linters, secret detection and diff-informed static analysis on changed lines. Keep this stage quick enough to run on every push. Our piece on data flow analysis for faster PR scans goes into how to keep this stage cheap.
  2. Classify the change. Is it new code, a refactor, a config change or an interface change? Interface and security-sensitive changes escalate.
  3. Expand to repository context for escalations. Trace callers, interfaces and dependencies for each changed symbol.
  4. Demand evidence for every finding. A claim like “possible data leak” is a hypothesis. A claim with the code path, the input that triggers it and a reproducible check is a finding.
  5. Gate on a calibrated score, not on a feeling. A merge gate needs a number that means something, and that means knowing how often the tool is right.
  6. Treat security fixes as changes that need review too. A patch that closes one hole can open another; see our eight steps for verifying security fixes in PRs.

Language-specific checks slot into step 1. For Python, five PR security checks that keep merges moving is a good starting list, and for credentials, why scanners alone are not enough for secrets in code explains what a diff-scoped pattern match can miss. If your team also maintains public-facing sites, a security code review habit pays off well beyond application repositories.

Measuring a diff-based reviewer honestly

A reviewer that flags everything has high recall and no usable precision. One that flags only the obvious has the opposite problem. Balancing the two is a tightrope, and the single number that captures it is F1, the harmonic mean of precision and recall.

Published work on security patch identification shows what reporting this looks like: the TMVDPatch multi-view learning approach reports an F1 score of 0.9001 and accuracy of 85.29%. Those figures belong to that system on its own dataset, and they are not a promise about any other tool. The habit worth copying is the reporting: precision, recall and F1, measured against real, confirmed defects.

When you evaluate any diff-based or AI reviewer, ask:

  • Is the score computed from confirmed defects, or from the tool’s own opinion of itself?
  • Does every finding come with evidence you can check yourself?
  • Does the tool read beyond the diff when the change warrants it?
  • Can the output feed a merge gate without a human translating it?

You should not have to take any vendor’s quality on faith, including ours.

Our perspective: the diff is where review starts

We built Veridical around a simple observation: the diff is the best map of where risk entered the codebase, and the worst description of what that risk touches. A reviewer, human or automated, that stops at the hunk boundary is reviewing the question the author asked and not the one production will ask.

So our reviews begin at the diff and then read repository context: callers, interfaces and dependencies. We publish a finding only when it is tied to concrete evidence and a reproducible check. If we cannot show you why something is a defect, it does not become a comment on your pull request, because every unverified comment lowers the trust a team has in all the others.

Every review ends with a calibrated advisory score, grounded in projections of real-defect F1, which you can wire into merge gating. We think a score should be something you can interrogate: what it measures, how it was derived and what a finding behind it looks like. We have written about one case where this mattered in how we would have caught a $100M+ defect that frontier models missed.

Our position is not that diff-based analysis is wrong. It is that a diff-scoped pass is the first stage of review and never the verdict.

— Łukasz

Where Veridical fits

If you want this workflow without assembling it yourself, Veridical reviews GitHub pull requests with verified findings, a detailed summary, inline evidence and an advisory score. Paid plans add deterministic checks, review history and repair workflows.

Pricing as published on our pricing page: the Standard plan is 39 USD per month and the Pro plan is 69 USD per month, and a Launch tier is also listed there. Open-source maintainers can use free reviews for open source, and our security and data handling page explains how private repositories are treated.

FAQ

What is diff-based analysis in code review?

It is analysis restricted to the lines a pull request adds, removes or changes. It gives fast, focused feedback, but it cannot see how those lines interact with code outside the diff unless the tool deliberately pulls in more context.

Is diff-based analysis enough for security review?

Often not. Guidance on diff-based review notes that it can miss broader security and logic issues, especially for small changes to authentication or secret handling. Use the diff to locate the change, then trace call chains and dependencies before you decide it is safe.

Which algorithm does git diff use?

Git uses the Myers algorithm by default. You can switch to minimal, patience or histogram with the --diff-algorithm option, which can change how hunks are split for the same underlying change.

Does scoping a scan to the diff really make it faster?

Yes, in documented cases. GitHub states that diff-informed CodeQL analysis can cut scan times by up to 10x compared with scanning the whole codebase. The trade-off is that anything outside the changed lines is not re-examined.

Sources

Diff-Based Analysis: What It Catches and What It Misses · Veridical.dev