Veridical.dev
Veridical14 min readArticle

Context-Aware Code Review: What It Takes and How to Verify It

← All articles

Context-Aware Code Review: What It Takes and How to Verify It

Geometric illustration of code context and verification

A context-aware code review evaluates a change against the code around it: the callers that depend on a modified function, the interfaces it implements, the dependencies it touches and the data that flows through it. A diff-only review sees what changed. A context-aware review asks what the change does to everything else. The distinction matters because the defects that cause data leaks and production incidents rarely sit inside the changed lines alone.

This guide explains what “context” has to mean in practice, how caller and call-graph analysis fit in, why context alone is not enough without verification, and how to evaluate any review process, ours included, against evidence.

Veridical
Verify Pull Requests With Evidence
Veridical reviews GitHub repository context and ties findings to concrete evidence, helping teams assess defects before merging code.
Visit Veridical

TL;DR

  • Context-aware review means analyzing callers, interfaces, dependencies and data flow around a pull request, not only the changed lines.
  • More context does not make a model trustworthy. A 2026 study of LLM-assisted security review measured false positive rates of 68.4% to 96.8% under neutral conditions, so findings need proof.
  • The strongest workflows separate finding from verification: a high-recall stage proposes, a validator pressure-tests against the real codebase.
  • Judge any tool by confirmed defects found and noise produced, not by how much code it reads.
  • Veridical reviews repository context and publishes findings only when tied to concrete evidence. Open-source projects get a free allowance.

Table of Contents

What “context” means in a pull request review

A pull request diff is a narrow window. It shows added and removed lines with a few surrounding ones. A reviewer, human or automated, who sees only that window must guess at everything outside it.

Context, in the sense that changes review outcomes, has four layers:

  1. Callers. Who invokes the function you modified, and what do they assume about its return values, error behavior and side effects?
  2. Interfaces and contracts. Does the change break an interface, a serialization format or an implicit contract that other modules implement or consume?
  3. Dependencies. What does the changed code call, and did the change alter what gets passed to it?
  4. Data flow. Where does a value originate, what validates it, and where does it end up? This is where authentication bypasses and leaks hide.

Guidance for security teams makes the same point: to find logic flaws such as authentication bypasses or insecure state assumptions, a tool must be fed architectural context such as call sites and dependencies. Without it, a model may flag secure idioms as risky while missing the vulnerability that actually ships.

A concrete example

Suppose a pull request changes a helper so that it returns null instead of throwing when a record is not found. The diff is three lines and looks like a harmless cleanup.

# before
def load_account(account_id):
    return db.get(account_id)  # raises NotFound

# after
def load_account(account_id):
    return db.find(account_id)  # returns None

A diff-only reviewer approves it. A context-aware review follows the callers and finds an authorization function that does if load_account(id).owner == user: with the exception previously acting as the guard. Now a missing record produces an attribute error, or in a looser language a fall-through path that skips the check. The defect is nowhere in the diff. It lives at a call site the author never opened.

That is the whole argument for context in one example: the lines that are wrong are not the lines that changed.

Impact analysis, call graphs and where they stop

Readers searching this topic often arrive with adjacent questions: how to run code change impact analysis, how call graph analysis works, how to find caller impact across a whole codebase. They are the same idea at different layers.

  • Call graph analysis maps which functions call which. From a changed function you walk upward to direct and transitive callers.
  • Caller impact analysis takes that walk and asks what each caller assumes about the changed behavior.
  • Codebase-wide impact analysis adds interface implementors, configuration, and tests that exercise the changed path.

The mechanics are well understood. The hard parts are the edges: dynamic dispatch, reflection, dependency injection, generated code and cross-service calls all break a naive graph. A static graph is an excellent way to decide where to look. It is a poor way to decide what is true, because reachability on paper does not prove that a path is exploitable or even executed.

That is why we treat the graph as a retrieval step. It tells the reviewer which callers, interfaces and dependencies belong in front of it. Whether a given caller is actually broken is a separate question that needs evidence. For the data-flow side of this, see our write-up on data flow analysis for faster pull request scans.

Why more context is not the same as more accuracy

It is tempting to conclude that if context helps, a model with a huge window and the whole repository will solve review. The evidence does not support that.

The 2026 study cited above, Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review, examined how the framing around a change affects automated reviewers. Two results stand out for anyone wiring a model into a pipeline:

The second finding deserves attention. A pull request description is context too, and it is written by the author, who may be mistaken or hostile. A tool that treats the description as truth inherits the author’s blind spot. The authors also point to an asymmetry: attackers can refine an attempt against a local clone of your pipeline, while defenders typically get one opportunity to catch an injection attempt in a PR. Their conclusion is to put independent verification stages in CI/CD rather than trust any single agent.

So the right framing is not “context versus no context.” It is context plus proof. Context raises recall, the chance that the defect is even in view. Proof controls precision, the chance that what you report is real.

Finder, critic, judge: separating recall from proof

The pattern emerging from security engineering teams is to stop asking one model to both suspect and convict. One published account describes a multi-stage design: a high-recall Finder, a Critic that tests each candidate against the actual codebase, and a Judge. The separation keeps noisy output from becoming a trusted decision and leaves an audit trail a human can inspect.

This maps directly onto what a context-aware review should do:

Stage Job Failure if skipped
Retrieve context Pull callers, interfaces, dependencies for the changed code Defects outside the diff are never seen
Propose findings Generate candidate defects, favoring recall Real defects are missed
Verify against the repo Check each candidate with deterministic checks and concrete evidence Noise gets reported as fact
Score and summarize Convert confirmed results into a decision signal Reviewers cannot act on the output

If a vendor cannot tell you which of these stages exist and what happens between proposal and publication, you are being asked to take the output on faith.

Treat AI findings as hypotheses

Security architecture guidance from 2026 is consistent here: AI-assisted review should not serve as a final certification of safety. It belongs in a layered defense with unit tests, static analysis and human oversight, and teams should treat its findings as hypotheses to verify against system behavior and logs. A related practitioner workflow reaches the same place from the other direction: AI reads intent and catches authorization and logic flaws that pattern scanners miss, but only when its output is handled with discipline.

If you want a broader grounding in verifying AI-generated security output as part of defense in depth, this overview of generative AI security practices is a reasonable companion read.

The practical consequence: a context-aware review that reports a vulnerability should show you the call site, the path and the reproducible check that supports it. A finding you cannot reproduce is a lead, not a defect. And a clean review is not a certificate. It is one control among several, and your tests, scanners and human reviewers still matter.

Three safeguards for verifying AI review findings

Human limits make context retrieval worth automating

There is a plain reason to automate context gathering. Humans are poor at it under time pressure. A large study of peer review at Cisco, summarized on one engineer’s blog, found that review effectiveness declines significantly after about an hour of effort, and that reviewers moving faster than 450 lines per hour had a below-average ability to find defects in 87% of cases. Chasing callers through a large codebase is exactly the work a tired reviewer skips.

Change size matters too. A controlled experiment on change decomposition in code review links splitting large changes into coherent pieces to more context-seeking by human reviewers and fewer wrongly reported issues. For automated review the implication is a division of labor: keep pull requests focused, narrow the review to the diff, and give the reviewer access to the repository context that the diff implies.

How to evaluate a context-aware review process

Whether you build it, buy it or run it by hand, test it with these questions.

  1. Does it read beyond the diff? Ask for an example where a finding cited a file the pull request never touched.
  2. Is every finding tied to evidence? Look for the code path, the call site and a check you could rerun.
  3. What is the noise rate? Count findings your team dismissed as wrong over a month. Study numbers on raw model output suggest this can be large without a verification stage.
  4. Does the pull request description influence the verdict? Try submitting a deliberately misleading description on a known-bad change. This is the adversarial framing test from the research above.
  5. Is the output a decision signal? A review that ends in a calibrated score can feed a merge gate. A pile of comments cannot.
  6. What does it not claim? Any tool that promises to certify safety is overselling.

A two-week trial on your own repositories answers most of these. Our guide to testing research-backed code review practices in two weeks lays out one way to run it.

Our perspective

We built Veridical around a simple observation: the cost of a review tool is not its subscription, it is the trust you spend on its output. Every false alarm teaches your team to skim, and every skimmed review lowers the floor under your process without anyone deciding to lower it.

Our perspective — overview diagram

So we start from the opposite end. A finding is published only when it is tied to concrete evidence and reproducible checks. Context is how we find candidates: we review repository context such as callers, interfaces and dependencies, not just the diff. Verification is how we decide what you see. Every review ends with a calibrated advisory score, and the score is built on a projection of real-defect F1, which balances recall against precision. Either one alone is easy to inflate. A tool can catch everything by flagging everything, or look precise by staying silent.

We would rather you not take that on faith. That is why findings carry inline evidence and summaries explain the reasoning, and why the score can be wired into a merge gate that you control. For how scores like this should be read, see our piece on why quality metrics need verified pull request reviews, and on finding what reviews miss, reducing false negatives with evidence.

We also hold a firm line on scope: Veridical is advisory. It sits alongside your tests, scanners and human reviewers, and does not replace them.

— Łukasz

Putting context-aware review into your GitHub workflow

A workable rollout does not need a platform migration.

  1. Start on a repository with real traffic. Quiet repos produce no signal.
  2. Keep pull requests focused. Smaller changes give both humans and tools a clearer question to answer.
  3. Compare against your last incidents. Replay a few pull requests that shipped a defect and see whether the review would have surfaced it with evidence.
  4. Read findings before trusting the score. Check that the evidence stands up, then decide whether the score matches your judgment.
  5. Gate gradually. Begin with the score as information, then require it for protected branches once you trust its calibration. Our walkthrough of GitHub CI quality gates covers the mechanics, and the six-step pull request review process shows the review loop end to end.

Secrets are a good first test of context, because a scanner sees a string while a reviewer must see where it flows. We discuss that gap in why developers need more than secret scanners.

Trying Veridical

Veridical is an evidence-based pull request review tool for GitHub repositories. It reviews repository context, publishes verified findings with inline evidence, and ends each review with an advisory score. Paid plans add deterministic checks, review history and repair workflows.

  • Open-source maintainers: there is a free allowance for open-source projects.
  • Teams: the Standard plan is 39 USD per month and Pro is 69 USD per month. A Launch tier also exists; see the pricing page for what each includes.
  • Security-conscious buyers: read how code and data are handled on our security page before connecting a private repository.

You can see how the review works on the AI code review for GitHub page.

FAQ

What is context-aware code review?

It is a review that evaluates a change against the surrounding codebase, including callers, interfaces, dependencies and data flow, rather than only the modified lines. The goal is to catch defects that appear at call sites or in downstream code the author never opened. It works best when each finding is verified rather than merely suggested.

Is a larger context window the same as context-aware review?

No. A large window lets a model read more text, but it does not decide which code is relevant or prove that a suspected defect is real. Research on LLM-assisted security review has measured high false positive rates even under neutral conditions, so retrieval of the right context and a verification step matter more than raw size.

How is this different from call graph analysis?

Call graph analysis maps which functions call which, and it is one way to choose the context for a review. A review adds the judgment step: whether a specific caller actually breaks under the change. Static graphs struggle with dynamic dispatch, reflection and generated code, so their output is best treated as a pointer for further checking.

Can an automated review replace human reviewers?

No. Current security guidance recommends using AI findings as hypotheses within a layered defense that includes tests, static analysis and human oversight. Automated review is most valuable when it removes the tedious work of tracing callers and leaves people to judge design and intent.

Sources