Building an Evaluation Suite

What an evaluation suite contains

Not one set. Four, with different jobs:

📅 Timeline

The core setA few hundred representative real cases with agreed acceptable outputs. Your specification, and the number you quote.

The hard setCases you know are difficult — ambiguous inputs, unusual formats, the tail from your hundred-case sample. Small, and the most informative when it moves.

The adversarial setPrompt injection attempts, requests to exceed scope, attempts to extract system instructions, offensive input. Tests the boundary rather than the capability.

The regression setSpecific past failures with their fixes. Every incident adds a case. Prevents re-breaking what you already fixed.

Teams typically build the first and stop. The regression set is the cheapest to maintain and the one that most reliably prevents embarrassment, because AI systems re-break old failures easily — a prompt change that fixes one behaviour can silently undo another.

Choosing the cases

The core set should be drawn from real usage, sampled to match its distribution. Two failure patterns to avoid:

Convenience sampling. Cases the team had to hand, which are systematically the ones already understood. A set built this way scores high and predicts nothing.

Interestingness bias. Over-weighting difficult cases because they are more engaging to discuss. This produces a set that understates real-world performance and makes every improvement look inadequate.

Sample randomly from real inputs, then add the hard cases as a separate set rather than mixing them in. Keeping them separate is what lets you report both "94% on representative traffic" and "61% on known-hard cases" — two true statements that together describe the product, where a blended number describes neither.

Size: a few hundred in the core set is usually enough to detect changes that matter. Precision beyond that costs more than it returns, and a set too large to review by hand stops being a product artefact and becomes infrastructure.

Grading outputs with no single right answer

The distinctive problem. For classification you compare to a label. For generated text there is no single correct output, and equality is the wrong test.

Three approaches, usually combined:

Criteria-based scoring. Define what makes an output acceptable and score against each criterion. For a drafted reply: does it answer the question asked, is every factual claim supported by the source, is the tone appropriate, is it free of invented specifics, is it the right length. Binary per criterion, then aggregate.

This is where product judgement enters most directly. The criteria are the product definition — whoever writes them decides what the system is trying to be.

Reference comparison. Collect what a competent human actually produced for the same input and compare on the criteria. Your edit history from the draft pattern supplies this for free.

Preference judgement. Present two outputs and ask which is better. Reliable for comparing versions, and does not require defining quality in the abstract — useful precisely when criteria are hard to articulate.

A note on using a model to grade model output: it is common, useful for scale, and must be validated against human judgement on a sample before it is trusted. An automated grader that disagrees with your own reviewers is measuring something, and it is not your product.

Keeping the set current

Evaluation sets decay. The product changes, users change, and inputs drift — module 8 covers the operational side.

Three habits:

Add cases from real failures continuously. Every user complaint, every escalation, every output you were embarrassed by.
Refresh a portion quarterly from recent real traffic, so the distribution tracks reality.
Re-examine the criteria, not just the cases. If the product's purpose has shifted, criteria written a year ago are measuring the old product.

The difference between a useful evaluation and a comfortable one is almost entirely in the criteria.

Weak criteria — the ones teams write first:

Is the response helpful?
Is the response accurate?
Is the tone appropriate?

These feel right and discriminate poorly. Two reviewers will disagree, the same reviewer will disagree with themselves next week, and nearly everything passes because "helpful" is generous.

Strong criteria are specific, binary and checkable by someone who was not in the room:

Does the response answer the question actually asked, rather than an adjacent one?
Is every factual claim traceable to a supplied source document? (Any claim not traceable fails this criterion.)
Does the response avoid stating a figure, date or name that does not appear in the source?
If the source does not contain the answer, does the response say so rather than infer?
Is the response under 200 words?

Note what changed. Each is answerable yes or no by a reader with the source in front of them. Two reviewers will agree. And critically, each maps to a failure mode you have actually seen — the fourth criterion exists because the system was inferring, and now you measure it.

That last point is the method: write a criterion for each way your system has actually failed. Criteria derived from observed failures discriminate; criteria derived from abstract quality do not.

One practical consequence worth planning for. Strong criteria make your scores look worse — a system that was "92% helpful" becomes "71% fully compliant across five criteria". That number is more useful and harder to present. Decide in advance that you want the honest one, because the pressure to revert to comfortable criteria arrives the first time you show the results to someone who was expecting good news.

❓ Knowledge Check

Why should hard cases be kept in a separate set rather than mixed into the core evaluation set?

📚 Flashcards1 / 6

Term

The four sets

Click to flip

Definition

The core set of representative real cases, the hard set of known-difficult ones, the adversarial set testing the boundary, and the regression set of past failures with their fixes.

Click to flip back

💡Key Takeaway

Build four sets with different jobs — core, hard, adversarial and regression — and keep them separate so you can report two true numbers rather than one meaningless blend. Sample the core set randomly from real traffic to avoid the convenience and interestingness biases. Write grading criteria that are specific, binary and derived from failures you have actually seen, because those discriminate where abstract quality criteria do not, and accept that honest criteria produce worse-looking scores.