Testing an AI agent: what unit tests can’t see

445 green tests, and the bug shipped anyway. A short question, just a reference, asked from an English interface: the assistant answered in French. The tests checked that the prompt said the right thing. Nothing measured what the model did.

This article is about how I evaluate the AI agent I put into production for maintenance teams: what I measure, how I read the results, and the bugs only the evals found.

Two nets, not one

Tests check the wiring: over 2,600 of them, with 99% branch coverage, lint, strict types and a security audit. If they are red, nothing ships. Evals measure behaviour, and they block nothing. They cost a few dollars, they are noisy, and the trade-off they inform (a little quality against a lot of latency, say) is a product decision, not a threshold.

What I measure

Retrieval, stage by stage

Retrieval is scored at each stage separately: words, meaning, hybrid, then reranking. That way, a number that moves names its own cause. On 77 real annotated questions, I track recall, MRR and nDCG@10. Correct answers are described by rules (“page 12 of the document whose title contains…”) rather than ids, so they survive a reindex.

Questions with no answer matter just as much: we measure whether their top result ranks above the median of real answers. Ranks, never scores, because two providers don’t score on the same scale.

Answers: deterministic first

Before any judge, dumb and infallible checks: does every cited reference exist in the context it was given? Is a source cited when one is needed? Is the expected figure, which only the SQL tool can produce, in the answer? Only then does a judge model answer three yes-or-no questions: does it answer, is it faithful to the sources, does it stay cautious when evidence is thin?

cited = set(REF_RE.findall(answer))
if cited:  # every reference the answer cites must exist in what it was given
    checks["grounded"] = cited <= set(REF_RE.findall(context))
if case.expect.get("citation") and context:
    checks["cited_something"] = bool(cited) or "](" in answer
if figure := case.expect.get("figure"):  # a count only the SQL tool can produce
    checks["right_figure"] = digits(figure) in digits(answer)

# the judge answers three yes/no questions, and fails closed
match = re.search(r"\{.*\}", raw, re.S)
if not match:
    return {}  # no verdict is never a pass

Yes or no, because a 1-to-5 scale invites arguing about a 3. The judge model stays the same across a comparison, so its bias cancels out. And an unreadable verdict is never counted as a pass.

Replay instead of re-searching

Tool outputs are captured once in production, then replayed. Two models get exactly the same context, with no VPN and no vector database. A tool missing from the recording answers “no results”: picking the wrong tool still costs.

Your turn

This bench reuses the e-bike support assistant from the hybrid search article. Pick a change and see what it breaks, what it fixes, and what it costs. Cases marked with a star are vetoes.

Reading an eval report

1. Judgement coverage, first. On the first run, reranking looked broken: its nDCG fell from 0.78 to 0.62. In fact, 39% of its top ten had never been judged, against 2.4% for hybrid, and an unjudged document counts as wrong. After 359 documents were adjudicated, coverage is printed before any score.
2. Vetoes, next. No invented reference (100% required), no leak on unanswerable questions after reranking. One miss, and the discussion ends.
3. The trade-off, last. Quality, cost, latency. A model change took cost per question from $0.0160 to $0.0131 and median latency from 8.5 to 5.6 seconds, matching or beating every check. Each model had only run once: I wrote that into the decision.

Bugs only the evals found

• Language. The fewer English words in the question, the more the answer drifted to French: 9 times out of 16 for a bare reference. The first fix itself failed the eval, the other way round. Putting the condition before the instruction solved it, and a test now checks the order of the prompt’s sentences.
• The vanishing tables. The first run of the extraction eval revealed that every table in every document was silently dropped. Values recovered: from 7 out of 18 to 16 out of 18 after the fix.
• The bench itself. A judge whose reasoning ate its whole token budget (five verdicts gone, which looked like a smaller sample), a token limit lower than production that cut answers before the mandatory notice, a capture where every tool failed. Hence 53 tests on the eval harness itself.

What I took away

• Tests block, evals inform. Don’t confuse the two.
• Score each stage separately, so a number that moves names its cause.
• Deterministic checks before the judge, yes-or-no criteria, and fail closed.
• Print coverage before the score.
• Treat the harness as production code: it lies too.
• Every bad answer reported becomes a case. The set grows with usage, and that is where its value is.