MODULE 5 · LESSON 3
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolExcessive Agency and Untrusted Output
The previous two lessons described the attack. This one is the defence, and it rests on a single reframing.
Stop asking how to prevent the injection. Assume it succeeds, and ask what it can then reach.
That reframing is forced on you by the first lesson: no filter is reliable, because no structural separation exists. So the security of an AI system is determined almost entirely by two decisions made when it was built. What is it allowed to do, and what happens to what it produces.
Excessive agency
The severity of a successful prompt injection is set by capability, not by cleverness.
Take the same assistant with three configurations and the same successful injection.
Read only, drafts for human approval. The injection causes it to draft something odd. A person reads the draft, notices, deletes it. This is a curiosity.
Read and send. The injection causes it to send data to an external address, or to send messages to your customers from a legitimate company address that passes every authenticity check. This is an incident.
Read, send, and act on business systems. The injection issues a refund, changes a record, deletes a file, or moves money. This is a serious breach.
Same attack. Same model. Same injection. Three completely different outcomes, decided by a configuration choice usually made for convenience.
The rule that follows: grant the minimum capability the job requires, and put a human approval step in front of anything irreversible or anything that leaves the organisation.
This is Module 2's least privilege applied to software, and it is more urgent here than for people, because a person's behaviour cannot be rewritten by a support ticket.
Model output is untrusted input
The second decision, and the one developers most often get wrong.
The rule is absolute and worth stating as a rule: treat everything a model produces exactly as you would treat text submitted by an anonymous stranger.
Not because the model is malicious, but because its output can be influenced by anyone whose text it read. If an assistant summarises a web page, the page's author has partial influence over the summary. That summary is now attacker influenced content.
So the normal precautions apply, unchanged:
- Output rendered in a web page must be escaped, or it is a cross site scripting vulnerability with an unusual delivery route.
- Output used in a database query must be parameterised, exactly as user input would be.
- Output passed to a shell command should not be, essentially ever.
- Output triggering an action needs validation against an allowed list, not merely a plausibility check.
LLM05 exists because the fluency and authority of the output makes people forget this. The text reads as though it came from a trusted colleague. It came from a system that read a stranger's document.
Agentic systems raise every stake
The direction of travel is toward systems that do not merely answer but act: reading mail and replying, browsing and purchasing, opening tickets, editing code, chaining tool calls together over many steps with limited supervision.
Everything in this module applies more sharply to those systems, for three reasons.
The input surface is larger. An agent that browses reads content from sources nobody vetted. Every page is potential injection.
Actions compound. A single wrong answer is recoverable. A sequence of actions taken on a wrong premise may not be, and by the time anyone reviews it the agent has taken twenty steps.
Supervision degrades. The value proposition of an agent is that you do not watch every step. That is precisely the property an attacker needs.
None of which is an argument against using them. It is an argument for the approval gate. The practical pattern that works: let the agent do the reading, the planning and the drafting freely, and require a human to authorise the small number of steps that are irreversible or externally visible. You keep nearly all of the productivity and remove nearly all of the worst outcomes.
Approval gates fail in a specific and predictable way. If every action requires confirmation, people click through without reading within about a week, and you have added friction while removing no risk. The gate has become theatre.
Designing one that survives contact with real work means being selective about what it stops.
Gate on irreversibility, not on importance. Sending an external email, moving money, deleting data, changing permissions, publishing something. Reading, searching, drafting and summarising should flow freely. Most of what an agent does is reversible, and gating it teaches people to click through.
Show the diff, not the intention. A prompt saying "the assistant wants to send an email, approve?" gets approved automatically. A prompt showing the actual recipient, subject and body, with anything unusual highlighted, gets read. The information has to be sufficient to make the decision, and specific enough to notice when something is wrong.
Make refusal cheap and normal. If declining means the work stops and someone has to start over, people approve to avoid the cost. Declining should return to a sensible state.
Set thresholds rather than a single rule. Small refunds proceed automatically, larger ones need a person, and anything above a threshold needs two. Uniform rules are either too slow for routine work or too loose for the rare dangerous case, and they get bypassed for the first reason.
Log approvals so they are reviewable. Not to blame the approver, but because reviewing what was approved is how you find out that the gate stopped meaning anything.
The underlying principle is the one from Module 2. A control requiring sustained human attention on a repetitive task will fail, because human attention on repetitive tasks reliably fails. Concentrate the attention on the few decisions that genuinely warrant it, and let everything else through.
A developer is concerned about prompt injection in a customer facing assistant. Which change most reduces the severity of a successful injection?
Capability determines severity
Click to flipThe damage from an injection is set by what the model may do afterwards, not by how the injection was crafted. The only reliable lever.
Click to flip backBecause no filter is reliable, stop asking how to prevent injection and assume it succeeds. Severity is then decided by two build time choices: what the system may do, and what happens to what it produces. Grant minimum capability and gate anything irreversible behind a human, and treat every model output as text from an anonymous stranger, escaping and parameterising it exactly as you would user input. Gate on irreversibility rather than importance, and show the actual action rather than the intention, or the gate becomes a button people click without reading.