Shipping Safely When the Ground Moves

Everything that changes behaviour is a version

In conventional software the code is the artefact. In an AI product, output depends on several things, and a change to any of them changes behaviour:

• The model, and its version.
• The prompt or system instruction.
• The retrieval configuration — what is fetched, how much, from where.
• The knowledge base contents.
• Parameters such as temperature and token limits.
• Pre- and post-processing rules.

All of these must be versioned together and deployable as a unit, because the question after any incident is "what was live when this happened?" and it must have a single answer.

The most common failure here is treating prompts as configuration rather than as code. A prompt edited directly in a production console, unversioned and unreviewed, is a change to your product's behaviour with no record — and prompts are edited far more often than models are.

Regression evaluation before release

Any change to any of the above triggers a run of the suite. The bar is not "did the score go up" but "did anything get materially worse?"

This matters because AI changes are frequently uneven. A prompt adjustment that improves handling of one input type routinely degrades another, and an aggregate score can rise while a category collapses. Report by category, not only in total.

A workable gate:

The regression and adversarial sets being zero-tolerance is what makes this workable in practice — it gives the team a clear, non-negotiable line without requiring judgement calls under release pressure.

When the provider changes the model

This is the situation with no equivalent in conventional software: your product's behaviour can change without you deploying anything.

A provider updates a model behind the same name, deprecates a version, or adjusts a safety layer, and your carefully tuned prompt now behaves differently.

The defences, in order:

Pin the version where you can. Many providers offer dated model identifiers. Use them, and upgrade deliberately rather than by default.

Contract for notice. Practical AI for Future Managers covers the term: notification of model change plus a testing window. Ask for it.

Run the suite on a schedule, not only on your own changes. A weekly regression run against a pinned production configuration detects a provider-side change you were not told about. This is the control that actually catches it.

Keep a rollback path. If upgrading a model degrades a category, you need to go back. That requires the previous configuration to be deployable, which requires the versioning above.

Staged rollout

Never move a change to all users at once. The staging that works:

• Internal first, on real inputs, with the team using it.
• A small percentage of live traffic, monitored against both offline categories and online signals.
• Increase in steps, with a defined rollback trigger at each.

Define the rollback trigger numerically before starting, exactly as with the stop condition in the managers course: "roll back if edit rate rises above X or refusal rate exceeds Y". A trigger written under pressure, after the change is live and someone's work is attached to it, is a negotiation rather than a control.

Give the rollout time to be observed. AI quality problems surface unevenly — a category that appears once a week will not show up in a two-hour canary. Match the observation window to how often your rarer input types actually occur.