Offline, Online and Human Evaluation

Three sources, three jobs

Offline evaluation runs your suite against a fixed set. It is fast, repeatable, and comparable across versions — so it is what you use to decide whether a change is an improvement.

Its limitation is absolute and worth stating plainly: it measures performance on the cases in the set. It cannot tell you about inputs you have not collected, and it cannot tell you whether users are better off.

Online signal comes from real usage. It tells you what actually happens, on the real distribution, with real users. It is slower, noisier, confounded by everything else in the product, and often ambiguous — but it is the only source that measures the thing you care about.

Human evaluation is people judging outputs. Expensive, slow, and the only method that reliably catches what neither of the others sees: the output that is technically compliant and wrong in a way no criterion anticipated.

The three are not alternatives. Offline for iteration speed, online for truth, human for the things you did not think to measure.

The gap between offline gains and product improvement

A recurring and expensive surprise: a change improves the evaluation score meaningfully and changes nothing for users.

Reasons this happens, all common:

• The gain is concentrated in cases users rarely hit. Your set is not distribution-matched, or the improvement lands on the tail.
• The improvement is below the threshold that changes behaviour. Going from 88% to 91% may not alter whether anyone accepts the output.
• Something else dominates. Latency, interface friction, or trust is the binding constraint, and accuracy was never what limited adoption.
• The metric moved, the experience did not. Especially with automated graders that reward properties users do not notice.

The discipline: treat an offline improvement as a hypothesis about the product, not as a result. It earns a place in an online test; it does not earn a claim.

Signals your surface can capture

Well-designed AI products instrument themselves. Most of these are free once the interface is right:

The distinction between edited and rewritten is the one teams most often collapse, and it carries the most information. A lightly edited draft is a success; a wholly replaced one is a failure that looks identical in a naive "was it used?" metric.

Two cautions. Acceptance is not satisfaction — users accept mediocre output when rejecting is expensive, which is why the friction symmetry from module 2 matters for measurement as well as for fairness. And implicit signals need validation: check periodically that your proxy tracks actual quality by comparing against human judgement on a sample.

Human review that is affordable

Human evaluation is worth doing and is usually organised badly — either too little to be informative or too much to be sustainable.

A workable shape:

• A small weekly sample, perhaps thirty to fifty outputs, reviewed against your criteria. Enough to detect meaningful drift, small enough to sustain indefinitely.
• Stratified rather than uniform. Over-sample low-confidence outputs, new input types and any category you recently changed. Uniform sampling spends most of its budget confirming the easy cases work.
• Rotating reviewers, including the product manager. Reviewer drift is real, and the PM reading outputs is the habit from module 1.
• Every disagreement examined. Where two reviewers score differently, the criteria are ambiguous — fix the criteria, and the fix improves every future evaluation.
• Findings go into the sets. Each interesting case becomes a core, hard or regression case. This is how the suite stays current at no extra cost.