Veridical.dev
Veridical18 min readArticle

F1 Score: 20 Misses Can Still Score 0.667 for Bug Detectors

← All articles

F1 Score: 20 Misses Can Still Score 0.667 for Bug Detectors

Geometric illustration of balanced detection and missed defects

The F1 score is the harmonic mean of precision and recall, ranging from 0 to 1, and it tells you how balanced a bug detector’s hits and misses are. For bug and vulnerability detection specifically, we recommend favoring recall-weighted measures such as Fβ while holding a minimum precision floor, because missed defects tend to cost more than extra triage time.

TL;DR:

  • For security checks, use F2 if costs are unclear, and keep precision near 75% so noisy alerts do not train reviewers to ignore findings.
  • One 2024 study found that realistic whole codebases cut precision by up to 95 percentage points and F1 by up to 91 versus curated benchmarks.
  • Publish raw TP, FP, and FN counts, per rule results, dataset version, and averaging mode; aggregate F1 can hide opposing failures across checks.
  • Track recall at the agreed precision floor across releases, and test with cross validation plus an untouched holdout set that resembles production code.
Veridical
veridical.dev
Ground Bug Detection in Evidence
Veridical reviews GitHub pull requests with verified findings, detailed summaries, and inline evidence to help teams assess defects before merging.
Visit Veridical

Table of Contents

What Are Precision, Recall, and the F1 Formula?

A true positive (TP) is a flagged issue that is a real bug. A false positive (FP) is a flagged issue that turns out to be fine. A false negative (FN) is a real bug the detector missed. Precision is TP divided by everything the detector flagged; recall is TP divided by everything that was actually a bug.

F1 combines both into one number using two equivalent formulas, confirmed in the scikit-learn documentation:

  • F1 = 2 × (precision × recall) / (precision + recall)
  • F1 = 2 × TP / (2 × TP + FP + FN)

For multiclass or multilabel problems, sklearn supports micro, macro, and weighted averaging, and its zero_division parameter controls what happens when a class has no predicted or true positives.

Walking Through a Confusion Matrix Example

Say a bug detector runs against a test set and produces TP = 30, FP = 10, FN = 20. We can calculate F1 two ways and confirm they match.

  1. Precision = 30 / (30 + 10) = 0.75
  2. Recall = 30 / (30 + 20) = 0.60
  3. F1 (ratio form) = 2 × (0.75 × 0.60) / (0.75 + 0.60) = 0.667
  4. F1 (count form) = 2 × 30 / (2 × 30 + 10 + 20) = 60 / 90 = 0.667

Pro Tip: When precision and recall are both zero, sklearn’s zero_division parameter decides whether the function returns 0, 1, or raises a warning; set it explicitly rather than relying on the default.

Both formulas agree, confirming the sklearn implementation treats them as mathematically identical, which matters when you are reconciling a hand calculation against a library output.

Beyond the raw number, sklearn’s average parameter (binary, micro, macro, weighted) and sample_weight argument change results on multiclass problems, so a reported F1 is meaningless without knowing which mode produced it. When you publish an F1 score internally, stamp it with the dataset version, the averaging mode, and the sklearn parameters used, so another engineer can reproduce the exact value later.

Why Recall Usually Outweighs Precision in Bug Hunting

Precision and recall serve different masters. Precision controls how much time your team burns triaging false alarms; recall controls how many real bugs slip through to production. For most bug-detection work, the cost of a missed vulnerability outweighs the cost of an extra review, which tips the balance toward recall.

Empirical work on search-based software testing backs this asymmetry directly. In one study, dropping recall by just 5% cost SBST tools roughly 7.5 fewer bugs found out of 420, while precision had a negligible practical effect once it cleared a reasonable floor. Recall sensitivity, in other words, is where the real damage happens.

  • Low precision creates noisy alerts that teams learn to ignore.
  • Ignored alerts get muted at the rule level, which quietly erodes effective recall over time.
  • A detector that looks fine in isolated testing can underperform once developers start filtering its output.

That feedback loop is the trap: a tool with poor precision does not just waste time, it trains your team to distrust it, which drags recall down in practice even if the model’s theoretical recall stays constant.

Pro Tip: Target recall first, but enforce a precision floor of around 75% (or whatever your team agrees on) so alerts stay trustworthy enough that engineers keep acting on them.

Tracking “recall at precision,” the recall achieved once precision crosses that floor, gives you a single operating point to monitor release over release.

Reading F1 Scores Without Getting Misled

A single F1 number without context is close to useless. Recent evaluations of LLM-based and industrial bug detectors have reported F1 scores ranging roughly from 0.36 to 0.86 depending on technique, prompting, and operating point, which means a bare “0.7 F1” claim tells you almost nothing about whether a tool is strong or mediocre for your use case.

Aggregate F1 also hides per-rule and per-check variation. A static analysis tool can report a respectable overall F1 while one specific rule produces almost entirely false positives and another produces almost entirely false negatives, a failure mode described directly in practitioner writing on precision and recall, which notes that two tools with opposite failure modes can land on the exact same F1.

  • Always publish raw TP, FP, and FN counts alongside the F1 ratio.
  • Break results down per rule or per check, not just as one blended score.
  • Stamp every reported number with the dataset and version it came from.
  • Prefer a precision-recall curve over a single point when you can generate one.

Benchmark results reported on curated corpora can look dramatically better than reality, with one 2024 study finding that precision and F1 dropped by up to 95 and 91 percentage points respectively when the same models were evaluated on realistic, whole-codebase datasets instead of curated ones.

Choosing Between F1 and Fβ for Your Risk Model

Standard F1 weights precision and recall equally, which is rarely the right call for defect detection. The Fβ formula generalizes it: Fβ = (1 + β²) × (precision × recall) / (β² × precision + recall). A β greater than 1 weights recall more heavily; a β less than 1 weights precision more heavily.

For security-relevant bug detection, an F2 score (β = 2) is often more defensible than plain F1, since missed vulnerabilities typically cost more than a false alarm a reviewer dismisses in seconds, a point echoed in practitioner guidance on static analysis metrics.

  • Pick β from your actual cost ratio of false negatives to false positives, not by default.
  • Keep a hard precision floor (commonly around 75%) regardless of which β you choose.
  • Instrument counts per check, not just a blended organization-wide score.
  • Track recall at your chosen precision floor release over release.

Pro Tip: If you cannot confidently state your FN-to-FP cost ratio, start with F2 as a reasonable default for anything touching security or data integrity, and revisit it once you have a few months of triage data.

What Counts as a “Bug” When You’re Scoring a Detector

F1 is only as meaningful as the label taxonomy behind it, and bug-detection classifiers typically get evaluated against a mix of categories that behave very differently. Null pointer dereferences and off-by-one errors tend to be mechanically verifiable and produce cleaner TP/FP labels. Security vulnerabilities such as injection flaws or insecure deserialization often require deeper reasoning about exploitability, which makes labeling slower and noisier. Concurrency bugs, like race conditions and deadlocks, are notoriously hard to reproduce, so ground truth for them is thinner than for simpler logic errors.

Performance regressions sit in their own category again: a detector might correctly flag a slow query pattern, but whether that counts as a “true positive” often depends on a threshold someone set arbitrarily during dataset construction. API misuse and type errors are comparatively easy to grade mechanically, which is part of why many published benchmarks lean on them for high reported scores.

Five bug categories with different evaluation challenges

When you read a vendor’s F1 claim, check which bug categories it was measured against. A detector tuned for a narrow slice, say, dependency vulnerabilities, can post an impressive F1 on that slice while performing far worse on logic bugs or performance issues it was never evaluated on. Treat an F1 score as a claim about a specific bug taxonomy, not a general statement about code quality coverage.

Why Ground Truth for Bug Detection Is Hard to Get Right

Every F1 score inherits the quality of the labels it was computed against, and labeling bugs is harder than it looks. Unlike image classification, where a human can usually agree instantly on what’s in a photo, deciding whether a flagged code pattern is a genuine bug often requires understanding intent, call-site context, and sometimes production behavior that isn’t visible in the diff.

Disagreement between human labelers is common, especially for security findings where exploitability is judgment-dependent rather than binary. A finding that two reviewers both think is “probably fine” can still be a real vulnerability under specific input conditions neither considered. That ambiguity shows up directly in F1 numbers: a different labeling pass can shift reported precision and recall meaningfully, even with the exact same detector output.

Dataset provenance compounds the problem, as detailed in methods to catch look ahead bias that ensure dataset integrity. Many public vulnerability-detection datasets are built from already-patched commits, which can bias what a detector learns to recognize toward patterns that happened to get fixed and documented, rather than the full population of real-world defects. That mismatch is part of why realistic, whole-codebase evaluation produces such different numbers than curated benchmark evaluation, as the sharp precision and F1 drops in recent vulnerability-detection research illustrate.

Until your label process is documented and auditable, treat any reported F1 as provisional, and prefer datasets and labeling protocols that are published alongside the score rather than taken on faith.

Validating a Detector Beyond a Single F1 Number

F1 is a summary statistic, and summary statistics hide detail that matters for a production decision. Cross-validation across multiple folds of your dataset shows whether a detector’s performance is stable or whether it’s an artifact of one lucky train-test split. A model that posts a strong F1 on one split and a mediocre one on another is not ready for a merge gate.

The confusion matrix itself deserves direct inspection rather than collapsing straight to a ratio. Looking at raw TP, FP, and FN counts, and breaking them down by bug type or rule, reveals whether errors cluster in one category (say, concurrency bugs) while performing well elsewhere. A precision-recall curve, rather than a single operating point, shows how a detector trades one error type for the other as its confidence threshold shifts, which is more informative than any single F1 value.

It’s also worth testing on a holdout set that was never touched during model development or threshold tuning, and ideally on a dataset resembling real production code rather than a curated benchmark. Comparing performance across both gives you an honest read on how much of your F1 is genuine signal versus overfitting to a convenient test set. None of this replaces F1, it contextualizes it, turning one number into a defensible claim about detector reliability.

Why Dataset Size and Quality Change What F1 Can Tell You

A reported F1 score is a point estimate, and like any estimate, it gets less reliable as the underlying sample shrinks. A detector tested against a few dozen labeled bugs can show wide swings in F1 from one evaluation run to the next, simply because a handful of label changes shifts the ratio significantly. Larger, well-curated test sets narrow that uncertainty and make the reported number more trustworthy as a predictor of real-world behavior.

Size alone is not enough. A large dataset built from a narrow slice of bug types, or skewed heavily toward one codebase’s conventions, will still produce an F1 that generalizes poorly outside that slice. Class imbalance compounds this: most codebases have far more clean code than buggy code, so a dataset that doesn’t account for that imbalance can make a detector’s F1 look stronger or weaker than it would on a representative sample.

Quality control matters as much as volume. Noisy or inconsistently applied labels inflate both false positives and false negatives in ways that are hard to separate from genuine detector errors. As we’ve noted elsewhere in this guide, evaluating on realistic, whole-codebase data rather than curated corpora exposes gaps that smaller, cleaner benchmarks paper over. When you’re deciding how much to trust a vendor’s reported F1, ask about sample size, class balance, and how labels were produced before taking the number at face value.

Why Dataset Size and Quality Change What F1 Can Tell You — overview diagram

Putting F1 Metrics Into CI and Pull Request Review

F1 becomes genuinely useful only when it’s tied to a workflow decision, not left as a reporting artifact. The most direct place to apply it is the pull request itself: tracking per-PR TP, FP, and FN counts against a known baseline lets a team see whether a detector’s performance is drifting as the codebase evolves, rather than relying on a one-time benchmark.

A practical pattern is to compute recall-at-precision on a rolling window of recent PRs, then feed that number into a merge gate alongside other checks. If recall-at-precision drops below an agreed threshold, that’s a signal to review recent rule changes or retraining before trusting the detector’s output on new code. Version-stamping every metric you publish, which detector version, which dataset, which averaging mode, keeps the comparison honest across CI runs.

It also helps to separate detector-level metrics from rule-level metrics inside the same pipeline. A CI dashboard that only reports an aggregate F1 can mask one noisy rule dragging down an otherwise solid detector, the same per-check blind spot practitioner analysis of static analysis tools warns about. Surfacing per-rule breakdowns inside the PR review itself, rather than only in a separate analytics tool, gives reviewers the context to judge whether a flagged issue is worth their attention before they even open the diff.

How F1 Connects to Security and Performance Goals

Developers rarely care about F1 for its own sake. What they care about is shipping fewer vulnerabilities and fewer performance regressions without drowning in false alarms, and F1 is only useful insofar as it reflects progress toward that goal. A detector optimized purely for recall will catch more real issues but may flood reviewers with enough noise that they stop reading flagged items carefully, undermining the very detection it was meant to improve.

This is why the recall-first-with-a-precision-floor approach matters operationally, not just mathematically. A security-relevant miss, an injection vulnerability that ships to production, carries a cost that dwarfs the minutes a reviewer spends dismissing a false positive. Weighting toward recall with Fβ reflects that cost asymmetry directly rather than treating both error types as equally bad, which plain F1 implicitly does.

Performance regressions follow a similar logic but with a different cost curve: a missed regression might not be catastrophic, but it compounds across releases if nothing catches it early. Framing detector evaluation around the business cost of FN versus FP, rather than around chasing a higher F1 in the abstract, keeps metric work tied to what actually matters, fewer vulnerabilities shipped, fewer regressions reaching users, and a review process developers trust enough to keep using.

Why Teams Get Metric Priorities Backward

Most teams treat F1 as the finish line instead of a diagnostic. The conventional wisdom, pick the detector with the highest F1, quietly rewards tools that balance precision and recall evenly, when bug detection rarely calls for an even balance. A tool with slightly lower F1 but meaningfully higher recall can prevent more real incidents than one optimized to look good on a single blended score.

What gets underestimated is how fast low precision erodes recall in practice. A detector’s published recall assumes every flagged issue gets reviewed. Once developers start ignoring a noisy rule, which happens quietly and without anyone updating the benchmark, the effective recall in production falls well below what the test set reported. The number on the slide deck and the number that matters on a live codebase diverge, and nobody notices until something ships.

The fix isn’t a smarter formula. It’s discipline: publish raw counts, stamp every score with its dataset and version, and treat F1 as one input to a precision floor and a recall target rather than a single verdict.

— Łukasz

Measuring and Improving F1 With Evidence-Based PR Review

Confusion-matrix counts are only trustworthy when every flagged item has been checked against what actually happened in the code, which is the starting point for our approach at Veridical. Each finding we surface comes with inline evidence rather than a bare flag, so the TP and FP counts behind any reported score reflect confirmed defects, not guesses, and the F1 projection attached to a pull request means something concrete.

Veridical

We also review repository context, callers, interfaces, and dependencies, rather than scoring a diff in isolation, which helps keep both false positives and false negatives closer to their true rates. Every review ends with a calibrated advisory score built on real-defect F1 projections, which teams can wire directly into a merge gate instead of relying on a static threshold that never adjusts.

  • Verified findings with inline evidence, so TP/FP/FN counts reflect confirmed defects.
  • Repository-context review that checks callers and dependencies, not just the diff.
  • A calibrated advisory score usable as a merge-gate input.

To put this guide’s recommendations into practice, start with our free tier for open-source projects, or check plan pricing, where Standard runs $39 per month and Pro runs $69 per month, to instrument a pilot and start tracking recall at your chosen precision floor.

FAQ

What is a good F1 score for a bug detector?

There’s no universal threshold, since reported F1 for bug and vulnerability detectors has ranged from roughly 0.36 to 0.86 depending on technique and evaluation setup. Judge a score against your own precision floor and recall target rather than a generic benchmark number.

Why does F1 matter more than accuracy for bug detection?

Bug datasets are heavily imbalanced, with far more clean code than buggy code, so accuracy can look high even when a detector misses most real bugs. F1, as defined in the scikit-learn documentation, focuses on precision and recall together and better reflects detection quality on imbalanced data.

Should I prioritize precision or recall for security bugs?

For security-relevant detection, recall usually matters more because a missed vulnerability tends to cost more than an extra false alarm a reviewer dismisses quickly. Evidence from search-based software testing shows recall drops have an outsized effect on bugs found, which is why an F2-style, recall-weighted metric is often more defensible than plain F1, per arXiv research on defect predictors.

How do I calculate F1 from a confusion matrix?

Use F1 = 2 × TP / (2 × TP + FP + FN), which gives the same result as the precision-recall ratio formula. With TP = 30, FP = 10, and FN = 20, for example, F1 works out to 0.667 by either method.

What does Veridical’s F1 projection score mean?

Our advisory score is a calibrated projection built on real-defect F1 metrics from verified findings, meant to support merge-gate decisions rather than replace human review. Current plan pricing is listed on our pricing page.

Sources