Drift, Silent Failure and Cost Control

AI systems fail silently

Ordinary software fails loudly. It throws an error, a page fails to load, someone notices within minutes.

An AI system that has degraded keeps running. It returns answers with the same confidence and the same latency. Nothing alerts. The only difference is that more of the answers are wrong, and since nobody was checking every answer, nobody knows.

This is the defining operational characteristic, and every monitoring decision follows from it: you have to go looking, because it will not tell you.

The forms of drift

Input drift is the most common and the easiest to detect: watch the distribution of what arrives, not only the accuracy on what you tested. A rise in exception volume is usually input drift announcing itself.

Concept drift is the hardest, because inputs look normal and the answers are simply less right. Only outcome measurement catches it.

Model change is why module 5 argued for a notification clause. Without one, your first indication is behaviour changing for no visible reason.

Usage drift is the one nobody monitors and it accounts for a surprising share of "the system got worse" reports. The system is unchanged; the process around it moved.

Monitoring that actually detects

Four mechanisms, in increasing order of value:

1. A fixed evaluation set, re-run periodically. Cheap and necessary. Its limitation is fundamental: it tells you how the system performs on last year's cases, so it cannot detect input drift by construction.

2. Input distribution monitoring. Track what arrives — document types, lengths, categories, sources — and alert on shifts. Detects input drift before accuracy visibly falls.

3. Sampled human review of live output. The most valuable and most often skipped. Pull a small random sample of live decisions each week and have someone competent check them. Fifty cases a week is 2,600 a year, which is ample to detect degradation and to keep your evaluation set current.

4. Outcome tracking. Did the decision turn out to be right, where you eventually find out — the flagged invoice really was wrong, the routed ticket did not get re-routed. The only mechanism that catches concept drift, and the only one measuring what the business actually cares about.

The re-routing example is worth noting as a pattern: the cheapest outcome signals are usually already in your systems, in the form of downstream corrections nobody thought to count.

Controlling cost without suppressing adoption

Usage-based pricing creates a tension: the thing you want most, adoption, is the thing that raises the bill.

Handle it with design rather than restriction:

• Track cost per unit of work, never total cost alone. Rising cost at flat unit cost is growth; rising unit cost is a problem.
• Ask what determines unit cost. How much context is sent, whether whole documents are passed or relevant sections, what is cached, which model size handles which task. These are engineering decisions with direct financial consequences, and asking about them is legitimate.
• Route by difficulty. Many requests do not need the most capable model. Sending easy cases to a cheaper path and hard cases to an expensive one is standard practice and frequently the largest available saving.
• Cache the repeated. If the same question is asked a thousand times, answering it once is both cheaper and faster.
• Set alerts, not caps. A hard cap that stops a production system mid-month is an outage. An alert at a threshold gives you a decision.

Do not solve a cost problem by restricting access. Suppressing adoption to reduce the bill destroys the benefit that justified the system — a trade nobody would approve if it were stated plainly, which is why it should be stated plainly whenever it is proposed.