Your AI Feature Works. Now Make Sure It Keeps Working.
Build a complete test suite for your AI feature in about 200 lines of Python: a set of examples with known answers, automatic checks, and a final verdict that fails your pull request when your AI gets worse. No frameworks.
In March 2026, Anthropic released three changes that made Claude Code worse. The changes landed on March 4, March 26, and April 16. All three slipped past code review, unit tests, end-to-end tests, and automated verification. For six weeks, users reported that Claude Code "felt less intelligent", and Anthropic's own postmortem says it plainly: "neither our internal usage nor evals initially reproduced the issues identified."
One of the three changes, a prompt edit meant to make responses shorter, actually passed their tests before it was released. A broader evaluation, run after the damage was done, showed a 3% quality drop the original tests never measured.
This is the most instructive AI failure of the year, and the lesson is not "even Anthropic messes up." The lesson is: your tests only protect what they actually check. Whatever your tests don't measure, you can't see. Anthropic's tests didn't fail because testing AI doesn't work. They failed because the tests didn't measure what changed.
And here's the uncomfortable part: if you have released an AI feature, a chatbot, an email sorter, a summarizer, you probably have no tests for it at all. Not weak ones. None. Someone says your feature "got worse last week," and you have no way to know if they're right, and no way to prove your fix made anything better.
This post fixes that. You will build a complete test suite for your AI feature in about 200 lines of Python: a set of examples with known answers, automatic checks that grade the model's output, and a final verdict that fails your pull request when quality drops. No DeepEval, no Ragas, no Braintrust, no dashboards. When you finish, your terminal will look like this when someone breaks your AI feature:
================================================================
id det judge total note
----------------------------------------------------------------
001 2 2 4 SOFT judge verdict: 2
002 2 1 3 FAIL judge verdict: 1
003 2 2 4 SOFT judge verdict: 2
...
REGRESSION: 2 row(s) passed in baseline but fail now:
002: 5 -> 3 (judge verdict: 1)
010: 5 -> 3 (judge verdict: 1)
Gate: FAILThat red Gate: FAIL line is the runner's final answer: the tests failed, so the change can't merge. It is what Anthropic didn't have for six weeks.
Why not just use a framework?
Before you object: DeepEval, Ragas, and LangSmith exist, and they are not bad products. But they don't solve the two problems that make this work hard:
- Writing down what "correct" looks like. Someone has to collect examples of your feature's work and say what the right answer is for each one. No framework does this for you.
- Writing the grading rules. Someone has to define when an answer earns full marks versus partial marks. No framework does this either.
Those two jobs are 80% of the work. If you outsource the remaining 20%, the runner, the scorecard, the comparison, to a paid service, you still have to do the hard parts. And now you depend on a vendor for code that is, honestly, just tests.
Because that's the secret: testing an AI feature is just testing. The answers are fuzzy instead of exact, but the shape is identical: fixed inputs, expected outputs, a way to score the result, and a final verdict that fails when quality drops. You already know how to write tests. This post adapts that knowledge.
What testing an AI feature looks like
A regular test says that add(2, 2) returns 4. Deterministic. Exact. One right answer.
An AI feature gives different answers each time you ask, so its tests have to handle three differences:
- Different answers each run. The same prompt can return different valid answers on different runs.
- Many right answers. "High urgency" can be expressed five ways; all are correct.
- Quality you can't compare with equals. Some differences, like tone or judgment, can't be checked by comparing two strings.
So instead of a test file, you build three pieces:
- A set of examples with known answers. Real inputs for your feature, each with the answer you expect.
- Checks that grade the output. Some checks are exact ("does the output contain these fields?"), some grade quality ("is this the right level of urgency?").
- A runner with a final verdict. It runs every example, prints a scorecard, and fails the build when quality drops.
That's it. Everything else, the dashboards and paid tools, is decoration on top of these three pieces. You will build all three in this post.
The demo app: an email sorter
To test something, you need something to test. You will build and then test an email sorter: it takes an email and returns the sender, subject, and an urgency rating (high, medium, or low).
This app is chosen deliberately. It has:
- Broken output you can catch with code: invalid JSON, missing fields, an urgency outside the allowed values
- Wrong judgment you can't catch with code: the model calls a "medium" email "high". The format is perfect. The decision is wrong.
- Many correct answers: many emails have defensible urgency ratings, which is exactly the real-world problem these tests need to handle
Here is the whole app:
# extractor.py
import json
import os
from dotenv import load_dotenv
from google import genai
load_dotenv()
EXTRACTOR_PROMPT = """You are an email triage assistant.
Given an email, return ONLY a JSON object with exactly these fields:
- "sender": the sender's email address
- "subject": the email's subject line
- "urgency": one of "high", "medium", or "low"
No markdown, no explanation, just the JSON object.
Email:
{email}"""
def make_client() -> genai.Client:
return genai.Client(api_key=os.environ["GEMINI_API_KEY"])
def parse_json(text: str) -> dict:
"""Parse the model's JSON, tolerating a markdown code fence."""
text = text.strip()
if text.startswith("```"):
text = text.strip("`").removeprefix("json").strip()
return json.loads(text)
def extract(email: str) -> dict:
"""Return {"sender", "subject", "urgency"} for the given email text."""
client = make_client()
response = client.models.generate_content(
model="gemini-2.5-flash",
contents=EXTRACTOR_PROMPT.format(email=email),
)
return parse_json(response.text)Thirty lines, including the imports. This is the app the rest of the post protects.
Step 1: Collect examples with known answers
Now you label data. This is the work no framework will do for you.
Save 15 emails with their expected outputs in golden.jsonl, one JSON object per line:
{"id": "001", "email": "From: billing@vendor.com\nSubject: Invoice #4471 overdue 30 days\n\nYour invoice #4471 is now 30 days overdue...", "expected": {"sender": "billing@vendor.com", "subject": "Invoice #4471 overdue 30 days", "urgency": "high"}}Fifteen examples is enough to catch real problems and small enough to label in one sitting. Include:
- Obvious cases: a production outage is
high, a newsletter islow - Borderline cases: a contract with a deadline, an audit request with a due date
- Tricky cases: a phishing email that shouts urgency but isn't from a real domain
The borderline examples matter most. They are where your model's judgment actually shows, and where problems surface first. When you collect these, write down why you chose each label. That reasoning becomes your grading rules in the next section.
Why one JSON object per line instead of a CSV file? Because emails contain commas, quotes, and newlines. This format handles all three without escaping pain, and one object per line makes changes easy to read in code review.
Step 2: Check the output structure with code
Some failures need no judgment. If the model returns invalid JSON, or omits the sender field, or says urgency: "extreme", you don't need an AI to tell you that's broken. Check it with code:
# scorers.py
URGENCIES = {"high", "medium", "low"}
def score_deterministic(row: dict, actual: dict) -> tuple[int, list[str]]:
"""Check structure: valid JSON, fields present, urgency in the enum."""
notes = []
score = 0
if actual is not None:
score += 1
notes.append("valid JSON: yes")
else:
notes.append("valid JSON: no (parse failed)")
return score, notes
if isinstance(actual, dict) and all(k in actual for k in ("sender", "subject", "urgency")):
score += 1
notes.append("all fields present: yes")
else:
notes.append("all fields present: no")
if isinstance(actual, dict) and actual.get("urgency") in URGENCIES:
score += 1
notes.append("urgency in enum: yes")
else:
notes.append("urgency in enum: no")
return score, notesThree checks, one point each. Every example can earn up to 3 here, and this check costs nothing to run, so broken output costs you nothing to catch.
Always run these checks first. They are free, instant, and catch the failure developers forget most often: the model returning something that isn't readable JSON at all.
Step 3: Grade the judgment with a second AI
Now the fuzzy part. When the model returns valid JSON but calls a medium-urgency email "high", code can't help you. You need judgment.
The obvious worry: using an AI to grade an AI sounds circular. It isn't, for three reasons:
- Give the grader clear rules, not vibes. The grader doesn't answer "is this good?" on a 1-10 scale. It picks 0, 1, or 2, where each number has a written definition:
JUDGE_PROMPT = """You are grading an email-triage system.
A golden example defines the correct urgency. The system produced its own urgency.
Compare the two using these anchors:
- 0: clearly wrong (e.g. flagged low when it's an outage or security incident)
- 1: defensible but questionable (a reasonable person could disagree)
- 2: correct or an equally defensible call on a borderline email
Return ONLY the digit 0, 1, or 2.
Golden example:
{golden}
Email:
{email}
System's urgency: {actual}"""- The grader must be a stronger model. The sorter runs on
gemini-2.5-flash. The grader runs ongemini-3.1-pro-preview. The grader should be at least as capable as the model it grades. - Clear rules beat open-ended scoring. This is not theoretical. We ran the same 15 examples through the same grader twice: once with the written 0/1/2 rules, once with a plain "rate correctness 0-10" and no rules. The rule-free grader gave 13 of 15 examples a 9 or 10, including answers that were flatly wrong. It rewarded confidence, not correctness. The grader with written rules scored honestly and flagged the two bad calls. Same model. The rules made the difference.
- Spot-check the grader against yourself. Run your grader once against answers you wrote, and see how often you agree. In building this post's test suite, an independent grading pass agreed with 14 of 15 human labels outright and graded the one disagreement, the model calling a compliance request "high" when a human said "medium", as "defensible but questionable." That agreement number is your evidence the grader isn't guessing. If your grader disagrees with you constantly, sharpen its rules before trusting the final verdict.
One more rule the grader enforces: it must return only a digit. If it writes sentences instead, score that example 0 and say so. A grader that can't follow instructions can't grade.
Step 4: Run everything and compare against last time
The runner ties it together. For every example, call the sorter, run both checks, print a scorecard, and compare against the previous run:
def run_row(row: dict) -> dict:
try:
actual = extract(row["email"])
except Exception as e: # the harness must survive any extractor failure
actual = None
error = str(e)
else:
error = None
det_score, det_notes = score_deterministic(row, actual)
judge_score, judge_note = score_judge(row, actual)
return {
"id": row["id"],
"det_score": det_score,
"judge_score": judge_score,
"total": det_score + judge_score,
"error": error,
}The rest of run_evals.py handles three jobs. First, the scorecard: one line per example, plus an overall percentage, so problems are visible at a glance. Second, a spending limit: it estimates what the run will cost before running, and refuses if it exceeds --cost-budget. This matters more than it sounds, because tests that silently burn money get deleted by whoever pays the API bill. Third, patience with flaky calls: grading calls can fail for reasons that have nothing to do with your feature, so it retries once before scoring an example as failed.
Two practical details worth stealing:
- Catch every error the model can throw. A crashing model call is a zero-score example, not a crashed test suite. Your tests must always finish.
- Save only the last score for each example. You don't need the full history, just what each example scored last time.
Watch the tests catch a real regression
Here is where this stops being theory.
Make a "reasonable improvement" to the sorter's prompt, the kind of change that gets released every day:
EXTRACTOR_PROMPT = """You are an email triage assistant.
Prioritize responsiveness: when in doubt, rate urgency HIGH so
nothing important is missed. Return ONLY a JSON object with:
- "sender", "subject", "urgency" (high|medium|low)
Email:
{email}"""It sounds sensible. Nothing crashes. Every row still returns valid JSON. Time to release it.
Then run the tests. For this capture, we also removed the formatting instructions from the prompt and turned up the randomness, the exact sloppiness that creeps in when someone "simplifies" a prompt:
================================================================
id det judge total note
----------------------------------------------------------------
001 2 2 4 SOFT judge verdict: 2
002 2 1 3 FAIL judge verdict: 1
003 2 2 4 SOFT judge verdict: 2
004 2 2 4 SOFT judge verdict: 2
...
010 2 1 3 FAIL judge verdict: 1
...
REGRESSION: 2 row(s) passed in baseline but fail now:
002: 5 -> 3 (judge verdict: 1)
010: 5 -> 3 (judge verdict: 1)
Gate: FAILThe structure checks caught part of it. The sloppy prompt made the model return "HIGH" in capital letters, which fails the format check. But the deeper damage was judgment: the model rated 10 of 15 emails "high", including the newsletter and the lunch invitation. The grader, scored against the written rules this time, flagged exactly the rows where the inflation crossed the line, and the comparison against the saved scores failed the build.
Notice what the rule-free grader did in the same run, when we scored the same outputs without written rules: it handed out 9s and 10s to almost everything, including the wrong answers. A grader without rules is nearly blind. The verdict only works because the rules give the grader a spine.
This is the Anthropic lesson in miniature: the regression was invisible to code review and to any test that checked "did it return valid JSON." It was only visible to a test that measured judgment, and it was only caught because the final verdict refused to pass it.
Make it automatic: fail the pull request, not the postmortem
A problem you catch locally is good. A problem that can't merge is better. Save the scorecard, commit it, and let your CI system compare every change against it:
# .github/workflows/evals.yml
name: evals
on:
pull_request:
paths: ["content/blog/code/evals-from-scratch/**"]
jobs:
eval-check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install google-genai python-dotenv pytest
- name: Unit tests (mocked, free)
run: pytest content/blog/code/evals-from-scratch/test_harness.py -q
- name: Eval check (fails the pull request if the AI got worse)
working-directory: content/blog/code/evals-from-scratch
env:
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
run: python run_evals.py --cost-budget 30000The pass/fail rule is deliberately strict: no example that passed before may fail now. Not "the overall score drops less than 2%". With 15 examples, one flipped example is a 6.7% swing in the overall score, and any percentage cutoff that loose is noise. Strict example-by-example checking is honest about the small size of the set: fifteen examples can't measure an average precisely, but they can absolutely tell you whether the thing that worked yesterday still works today.
That is also why growing your example set is the single best investment once this test suite exists. Every real problem you find in production is a future test example, labeled with the correct answer. Your tests get smarter every time something breaks.
What to do next
You now have the full loop: examples with known answers, automatic checks, a runner, and a final verdict. Where to take it:
- Grow your example set. Fifteen examples catch obvious problems. Fifty catch subtle ones. Every bug a user reports is a new example waiting to be labeled.
- Keep sharpening your grading rules. The grader is only as good as its rules. When the grader disagrees with you, the fix is usually a clearer rule, not a bigger model.
- Make important mistakes count more. A missed security email should cost more than a misfiled newsletter. Keep it simple: double weight on examples where failure is expensive.
- When to pay for a framework. If you need to share test results across teams, track changes to your example set over time, or route failures to human reviewers, Braintrust or LangSmith earn their keep. Until then, they are a dashboard for a folder of files you could own outright.
The frameworks don't solve the two hard problems. You just did them: 15 labeled examples and one page of grading rules. The 200 lines around them are yours to read, fix, and extend.
Build this for your own AI feature, and you'll never again have to answer "did it get worse?" with a shrug. The scorecard already knows.
Founder of DevGuild. I build tools for developers and write about Python, AI, and web development.
@RealDevGuild