Testing prompt and agent changes with a small regression suite
Thirty well-chosen cases, deterministic checks first and a model judge last, run several times per case. A practical eval setup a single developer can maintain.
This is a design review of one small system: the regression suite that sits next to a prompt-driven feature. To keep it concrete, take an agent that reads invoice text and returns a JSON object with a vendor, a total and a currency. It is one prompt, one output schema and a handful of tool calls, owned by one developer who changes the prompt every week or two. The question is what the smallest suite looks like that tells that developer whether a change made things worse, and where such a suite stops being trustworthy.
It builds on the habits in Writing specs and plans an AI agent can execute and Claude Code hooks, skills and memory, which cover how to instruct an agent. This one covers how to know whether a change to those instructions helped.
Requirements the suite has to meet#
Reading outputs by eye fails in a predictable way: any change looks good on the example that motivated it, and regressions live on the examples you did not look at. Outputs also vary between runs, so one good run proves little, and a model update behind the same name changes your prompts without a diff. The suite has to answer those failures while staying small enough to survive.
- It runs before every change to a prompt, a tool description, an instruction file or the model setting, so it must be cheap in money and minutes.
- One person maintains it, so it is files in the repository: no dashboard, no database.
- Verdicts are pass or fail decisions that can be explained, not scores to argue about.
- Each case runs several times, and the suite compares counts, not single outcomes.
- Cases come from real tasks and real failures. After a few months the suite is a record of what has gone wrong.
That rules out large public benchmarks (wrong question) and, for one developer, hosted evaluation platforms (more than needed). It leaves fixed case files with code checks, and a model as judge only for properties a script cannot see.
A case is a small JSON file#
The fields are a convention of this design, not a standard. Keep what your checks need.
{
"id": "extract-invoice-total",
"why": "Guards against returning the subtotal when a discount line is present",
"input": "Invoice text goes here, including a discount line and a tax line.",
"checks": [
{ "type": "json-schema", "schema": "invoice-result" },
{ "type": "equals", "path": "total", "value": 118.4 },
{ "type": "not-contains", "value": "I cannot" }
],
"runs": 5,
"minPassRate": 0.8
}The why field matters more than it looks. When the case fails six months later, it says what the case guards against, so you can decide whether the failure matters. Every real failure in production becomes a case before the fix is written.
The runner#
The runner calls a function you supply, runAgent, so it works for a single model call, a tool-using agent or a command-line tool. It runs each case runs times, counts how many runs passed every check, writes integer counts, and exits non-zero if any case falls below its own minPassRate.
Two details are worth getting right. The json-schema check has to validate for real: a check that cannot fail is worse than no check, so the schema is compiled with Ajv from evals/schemas/. And minPassRate is turned into a whole number of required passes once, so no later comparison touches floating-point rates.
import { readdir, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import Ajv from "ajv";
import { runAgent } from "./agent"; // yours: call the model, agent or CLI, return final text
type Check =
| { type: "equals"; path: string; value: unknown }
| { type: "contains"; value: string }
| { type: "not-contains"; value: string }
| { type: "json-schema"; schema: string };
interface Case {
id: string;
why: string;
input: string;
checks: Check[];
runs: number;
minPassRate: number;
}
const ajv = new Ajv({ allErrors: true });
const validators = new Map<string, ReturnType<typeof ajv.compile>>();
async function validator(name: string) {
let v = validators.get(name);
if (!v) {
const schema = JSON.parse(await readFile(join("evals/schemas", `${name}.json`), "utf8"));
v = ajv.compile(schema);
validators.set(name, v);
}
return v;
}
function parseJson(output: string): unknown {
try {
return JSON.parse(output);
} catch {
return undefined;
}
}
function getPath(obj: unknown, path: string): unknown {
return path.split(".").reduce<unknown>((o, k) => (o as Record<string, unknown> | undefined)?.[k], obj);
}
async function passes(check: Check, output: string): Promise<boolean> {
switch (check.type) {
case "contains":
return output.includes(check.value);
case "not-contains":
return !output.includes(check.value);
case "equals":
return getPath(parseJson(output), check.path) === check.value;
case "json-schema": {
const data = parseJson(output);
return data !== undefined && (await validator(check.schema))(data) === true;
}
}
}
async function main() {
const dir = "evals/cases";
const files = (await readdir(dir)).filter((f) => f.endsWith(".json"));
const results: Record<string, { passes: number; runs: number }> = {};
let belowFloor = false;
for (const file of files) {
const c: Case = JSON.parse(await readFile(join(dir, file), "utf8"));
let ok = 0;
for (let i = 0; i < c.runs; i++) {
const out = await runAgent(c.input);
const verdicts = await Promise.all(c.checks.map((chk) => passes(chk, out)));
if (verdicts.every(Boolean)) ok++;
}
results[c.id] = { passes: ok, runs: c.runs };
// The small epsilon stops 0.07 * 100 = 7.000000000000001 from rounding up to 8.
const required = Math.ceil(c.minPassRate * c.runs - 1e-9);
const flag = ok < required ? "BELOW FLOOR" : "ok";
if (ok < required) belowFloor = true;
console.log(`${c.id}: ${ok}/${c.runs} (needs ${required}) ${flag}`);
}
await writeFile("evals/latest.json", JSON.stringify(results, null, 2));
process.exit(belowFloor ? 1 : 0);
}
main();If runAgent drives Claude Code, the headless mode fits. claude -p runs non-interactively, --output-format json returns the answer in a result field with total_cost_usd (a client-side estimate that can differ from your bill), and --json-schema returns schema-conforming output in a structured_output field. Add --bare for evaluation runs: it skips hooks, skills, plugins, auto memory and CLAUDE.md discovery, so the same configuration runs on every machine, and it needs ANTHROPIC_API_KEY in the environment because it does not use a subscription login. One caveat from the documentation applies directly to the suite: if a run ends without valid structured output, the response has no structured_output field at all, so the wrapper must treat an absent field as a failed run, not as an empty success.
#!/usr/bin/env bash
# Prints the structured answer for the prompt in $1, or nothing if the run produced none.
set -euo pipefail
claude --bare -p "$1" \
--output-format json \
--json-schema "$(cat evals/schemas/invoice-result.json)" \
| jq -c '.structured_output // empty'Empty output fails the json-schema and equals checks, which is the right outcome. Keep the schema check even though Claude Code enforces the schema for you: the same suite should be able to run against any other backend, and a case that quietly stops checking structure when the backend changes is a hole.
The baseline is a file of counts#
The baseline is evals/baseline.json in the same shape as latest.json, committed to the repository. The comparison works on integer pass counts only. Comparing rates with a tolerance is a trap here: in JavaScript 0.8 - 0.2 is 0.6000000000000001, so a rule meant to tolerate one lost run out of five (a drop of 0.2) would flag it anyway. Counts avoid the whole class.
import { readFileSync } from "node:fs";
type Result = { passes: number; runs: number };
const base: Record<string, Result> = JSON.parse(readFileSync("evals/baseline.json", "utf8"));
const now: Record<string, Result> = JSON.parse(readFileSync("evals/latest.json", "utf8"));
// Flag a loss of two or more passes. One lost run is within normal variation.
const MARGIN = 2;
let failed = false;
for (const [id, cur] of Object.entries(now)) {
const before = base[id];
if (!before) {
console.log(`NEW ${id}: ${cur.passes}/${cur.runs}`);
continue;
}
if (before.runs !== cur.runs) {
console.log(`RERUN ${id}: run count changed (${before.runs} -> ${cur.runs}); re-baseline`);
failed = true;
continue;
}
if (cur.passes <= before.passes - MARGIN) {
console.log(`WORSE ${id}: ${before.passes}/${before.runs} -> ${cur.passes}/${cur.runs}`);
failed = true;
} else if (cur.passes >= before.passes + MARGIN) {
console.log(`BETTER ${id}: ${before.passes}/${before.runs} -> ${cur.passes}/${cur.runs}`);
}
}
process.exit(failed ? 1 : 0);When you accept a change, copy latest.json over baseline.json in the same commit, so the commit holds the change and the evidence for it.
What five runs can and cannot tell you#
Treat each run as a coin that lands on pass with some unknown probability p. Five runs give a count out of five, and the arithmetic is the binomial distribution. A few values show how little it supports:
- A case that truly passes 90 percent of the time shows 5 of 5 only 59 percent of the time. A perfect record is weak evidence of a high rate, and a baseline recorded on a lucky run is a sample, not a fact.
- If p falls from 0.9 to 0.6, a real regression, the "lose two or more passes" rule fires with a count of 3 or fewer. With p at 0.6 that happens 66 percent of the time. One in three real regressions of that size slip through on any single run.
- If p falls to 0.8, the same rule catches it 26 percent of the time. Small drops are effectively invisible at five runs.
- With p unchanged at 0.9 and a baseline of 5, a false alarm (3 or fewer) happens about 8 percent of the time.
So five runs detect big regressions and flag some noise; they do not measure small ones. Where it matters, add runs for that case alone. At 20 runs, a drop from 0.9 to 0.6 leaves 15 or fewer passes 95 percent of the time, while an unchanged 0.9 does so 4 percent of the time. The honest version of the suite's promise is: large, consistent breakage is caught cheaply, and anything finer needs more runs on the specific case. If a case flips between 2 of 5 and 5 of 5 across identical runs, the case or the prompt is unstable, and the fix is to tighten the check or the instruction, not to widen the margin.
A judge for what code cannot see#
Some properties no script can check: whether a summary is faithful, whether an explanation is clear. For those, a second model call grades the output against a short rubric, under four rules.
- Use it last and rarely, only for the checks code cannot do.
- Ask a narrow question with a yes or no answer and a quoted justification: "Does the summary state any fact not present in the source? Answer yes or no, then quote the fact." That beats a score out of ten.
- Calibrate it against your own judgement. Read twenty graded outputs yourself, and if you disagree often, fix the rubric before trusting the grader. Model judges tend to prefer longer, more confident answers.
- Keep the judge fixed. Changing the judge model changes the instrument, so re-baseline everything.
Agents that act#
For an agent that edits files or runs commands, the output text matters less than the effect, so the checks look at the effect. Did the tests that should pass now pass? Did the diff touch only the allowed paths? Did it avoid a forbidden action such as deleting a file or calling a paid API? Run each case in a fresh copy of a fixture repository so one run cannot contaminate the next, and stub the network, timestamps and random identifiers in the fixture, because environment noise produces failures unrelated to your change.
Running it, and what it costs#
Run the suite by hand before changing an instruction file or a prompt, and in CI on pull requests that touch those paths, with a path filter so unrelated changes skip it. The cost is cases times runs times the price of one call, plus judge calls: thirty cases at five runs is 150 calls per full run. Whether that is cents or dollars depends on the model and the size of each case, so check current pricing and print the total from the runner so you notice when it creeps up. A smoke subset of five cases covers fast iteration.
Where the design stops being trustworthy#
- Overfitting. Tune a prompt until every case passes and you may have taught it your cases. Keep a few held-out cases that you do not look at while tuning, and keep adding new real failures.
- Flaky checks. A check that fails at random teaches people to ignore failures. Fix or delete it, and prefer structure, required facts and forbidden content over exact wording.
- Stale cases. A case guarding behaviour you deliberately changed should be updated or removed; the
whyfield tells you which. - A green run is a smoke alarm, not an inspection. It means the cases you wrote still pass. It does not mean the system is correct.
- Taste. If the output is judged by taste, as in creative writing, a suite measures the wrong things. Keep a folder of approved examples, compare by reading, and call it a review, not a test.
- Too early. While you are still exploring what the agent should do, cases freeze decisions you have not made. Wait until there are a handful of behaviours you would be upset to lose.
Related articles
All articles →Hook exit codes that do not block, rules that never load, skills that abort and memory that stays on one machine: the failure modes and the fix for each.
11 min
A layered CLAUDE.md, tight permissions, hooks, skills and scoped MCP servers: the setup that keeps Claude Code safe and cheap across dozens of repositories.
13 min
A Firestore rules deploy replaces everything at once, with no canary. The traps in the rules and the release path, and the pipeline that catches them.
11 min