Reliability Engineering for AI Products

The failure modes you must handle

Model APIs fail in ways ordinary services do not, and each needs explicit handling:

The last is where module 3's scheduled evaluation earns its cost — it is the only mechanism that detects the failure with no error attached.

The fallback ladder

A single fallback is usually insufficient. Build a ladder, descending gracefully:

1. Retry with backoff. For transient failures and rate limits. Bounded — retries amplify congestion during an incident.
2. Failover to a second region or endpoint of the same provider.
3. Failover to a different model, possibly smaller or cheaper. Quality drops; the product works. This requires your evaluation to have measured the fallback model, or you are shipping unknown quality under stress.
4. Non-AI fallback. The rules-based path, cached responses, templates — the pre-AI mechanism module 2 warned against deleting.
5. Honest unavailability. A clear message and a preserved user context, so nothing is lost.

Descend explicitly, and tell the user when quality is degraded where it materially affects them. A silently worse answer is worse than an acknowledged one.

Retry, carefully

Retries are the most commonly mishandled part of this, and mishandling them turns a partial degradation into a full outage.

• Always bound them. Unlimited retries during a provider incident generate load exactly when the provider is struggling.
• Use exponential backoff with jitter. Without jitter, every client retries in synchronised waves.
• Do not retry non-retryable failures. A refusal or a malformed request will fail identically the second time and costs money each attempt.
• Budget retries against the user's patience, not only against the provider's limits. Three retries within a latency budget the user will not wait through is a failure with extra steps.

What counts as an AI incident

Broader than conventional outage definitions, and worth writing down before you need it:

• The service is unavailable or materially degraded — conventional.
• Output quality has dropped materially — detected by scheduled evaluation or by a spike in edit and override rates.
• The system produced something harmful or seriously wrong that reached a user.
• A safety or scope boundary was breached — the system did something it should refuse.
• Data went somewhere it should not — module 6.

The middle three have no conventional equivalent and are frequently missing from incident definitions, which means nobody declares them, which means they persist.

Response requires the same shape as any incident practice: a way to detect, a way to report, a decision on whether to disable the feature, an assessment of who was affected, and a record. Disabling the AI feature while keeping the product working is exactly why the degradation design above matters.

Commitments you can meet

Be careful what you promise:

Availability is promisable, because it depends on your architecture and your fallbacks.

Latency is promisable at a percentile, if you have measured it and control the path.

Accuracy or quality is not promisable as a guarantee, and you should resist pressure to do so. What you can offer is what the managers course recommended asking of vendors: measurement and reporting against an agreed metric, with a remedy if it degrades materially. That is honest, deliverable, and more useful to a customer than a guarantee nobody could enforce.

Where a customer insists on a quality commitment, the productive move is to define it narrowly and measurably on a fixed evaluation set you both agree to — which turns an unbounded promise into a specific, testable one.