4.1 Explicit criteria to improve precision
Specific criteria beat vague instructions
| Vague — does not work | Explicit — works |
|---|---|
| "Check that comments are accurate" | "Flag a comment only when the claimed behaviour contradicts the actual code behaviour" |
| "Be conservative" | "Report: null-dereference risks, unhandled promise rejections, SQL built by string concatenation. Skip: naming, formatting, patterns that are locally consistent" |
| "Only report high-confidence findings" | Per-severity definitions with a concrete code example for each level |
"Be conservative" and "only report high-confidence findings" do not improve precision. They are confidence-based filtering, and the model's confidence is not calibrated. Only specific categorical criteria move the false-positive rate.
False positives destroy trust asymmetrically
A single high-false-positive category undermines developer confidence in the accurate categories. The remediation the objectives name: temporarily disable the high-false-positive category to restore trust while you improve its prompt — do not leave it running and ask reviewers to ignore it.
Consistent severity classification
Define severity with concrete code examples per level:
## Severity
**critical** — data loss, auth bypass, or unhandled money mutation.
e.g. `await refund(amount)` with no idempotency key.
**major** — incorrect behaviour on a supported input.
e.g. `parseInt(v)` where `v` may be `"0x10"`.
**minor** — correct but fragile; will break on a plausible future change.
e.g. index-based access into a sorted array whose comparator is configurable.
Quiz
- Question 1
Your automated review posts about 40 findings per pull request and developers dismiss 31 of them, mostly comments of the form "consider extracting this into a helper" and "this variable could have a more descriptive name". The prompt currently says to report anything that could be improved and to be conservative. Which change most improves precision?
- Question 2
Six review categories run on every pull request. Five have false-positive rates between 4% and 9%. The sixth, comment accuracy, sits at 61%, and developers have begun ignoring all inline comments, including findings from the accurate five. Rewriting the comment-accuracy criteria properly will take about a week. What do you do now?
- Question 3
Your review’s comment-accuracy category currently asks Claude to check that comments are accurate. It flags any comment whose wording has drifted from the code’s current shape, including "returns the user" above a function that returns a User object. Which criterion cuts the false positives without losing the genuine finds?
- Question 4
Reviewers disagree with the severity label on roughly a third of findings, and the same class of issue comes back as "high" on one pull request and "medium" on the next. The prompt defines each of the three levels in a single sentence. What produces consistent classification?