Prompt injection and what it means for your AI feature

The question to ask before you ship an AI feature is not whether the model can be talked into saying something embarrassing. It is what the model is permitted to do while it is being talked into it. Prompt injection is the attack, it has no fix, and it has held first place in the OWASP Top 10 for LLM Applications in every edition, including the 2026 list published in August. You cannot prompt your way out of it. You can only decide in advance what a successful one costs you.

Instructions and data arrive on the same channel

A language model receives one stream of tokens. Your system prompt, the user’s question, the document you retrieved, the body of the email a tool fetched: by the time the model reads them they are the same kind of object, separated only by position and by whatever labels you wrapped around them. Which text counts as an instruction is a judgement the model makes, not a boundary the runtime enforces. That sits underneath the prompt, which is why a better prompt cannot remove it.

Simon Willison named the class in September 2022, after Riley Goodside demonstrated it publicly, choosing the name because the mechanics looked like SQL injection. The analogy is where most buyers get misled. SQL injection has a real fix: a parameterised query sends the statement and the values over separate channels, and the driver never parses one as the other. There is no parameterised query for a language model. Until somebody ships that separation, everything below is containment rather than repair.

The standards bodies cut the attack by channel. NIST’s taxonomy (AI 100-2e2025, March 2025) and MITRE ATLAS both separate direct injection, where your user types the payload, from indirect injection, where your user is innocent and the payload sits inside content your system fetched. ATLAS carries the pair as sub-techniques of AML.T0051. Greshake and colleagues gave the indirect case its first systematic treatment at the ACM AISec workshop in 2023, and it is the shape that should worry a product team, because it needs no hostile user.

Which leaves one fact to carry the rest of this article: a model reading attacker-controlled text is reading instructions from the attacker, and the only question is what those instructions can reach.

Where the untrusted text actually gets in

“Untrusted” does not mean “from the public internet”. It means any text whose author is not the party you are acting for. The list runs longer than teams expect.

• Retrieved documents. Anything in a retrieval index that a customer, vendor or contractor uploaded. If anyone outside your company can write to the index, so can an attacker.
• Inbound tickets and email. The body is composed by a stranger, and it is the exact text you are asking the model to act on.
• Fetched pages and search results. Including one that was fine when you tested it.
• Files and their edges. OCR output, PDF metadata, a spreadsheet cell, an EXIF field, a filename. A file named invoice-ignore-previous-instructions.pdf is a legal filename, and a document pipeline reads all of it.
• Another model’s output.Chain two calls and the first one’s output is untrusted input to the second, laundered into an internal-looking channel.
• Your own database, if users write to it. Profile fields, organisation names, note bodies, ticket comments, an account display name. Text that arrived six months ago is still attacker-controlled text.

There is an hour of whiteboard work here worth more than any tooling purchase: list every string that reaches the model’s context and, beside each, who authored it. If you cannot finish that list, you are not ready to discuss mitigations, because you do not know where the perimeter is.

Rude output is not the risk. Capability is.

Most buyer conversations about AI safety are about tone: will it say something offensive, will it invent a refund policy. Those are product problems. The security problem is the three below.

The tools it can call

An injected instruction is only as dangerous as the most dangerous tool in the model’s list. A summariser holding no tools is a content problem. The same model holding send_email, issue_refund or a general-purpose run_query is an execution surface with a friendly interface in front of it. The 2026 OWASP list puts Excessive Agency third, ahead of supply chain and poisoning.

The data it can read

The question is not what the current user may see. It is what the credential the model holds may see. A service account with organisation-wide read access, given to a model reading a document one customer uploaded, is a cross-tenant leak waiting for one paragraph. The tenancy model underneath decides whether that is a configuration mistake or an architectural one.

The actions that do not reverse

Money moving. An email leaving. A record deleted, a permission granted, a webhook fired at a third party. And exfiltration, which teams miss because it needs no tool called send: if your interface renders model-authored markdown, an image tag whose URL carries the data is an outbound channel no tool inventory lists.

That is the shape of EchoLeak, CVE-2025-32711, rated CVSS 9.3, the zero-click issue Aim Security disclosed in Microsoft 365 Copilot in June 2025. A crafted email arrived, Copilot read it while answering an unrelated question, and instructions inside it moved data out of the user’s context in a URL. Microsoft patched it server side and said it had not been exploited in the wild. The lesson is structural, not about one vendor: a rendering surface is an egress path.

Meta published the cleanest framing of the triangle in October 2025 as the Agents Rule of Two. Within a session, an agent should satisfy no more than two of: processing untrustworthy input, holding access to sensitive systems or private data, and being able to change state or communicate externally. Where a task needs all three, their guidance is that it should not run autonomously. A constraint rather than a proof, and still the most useful sentence a non-specialist can take into a design review.

The mitigations, ranked by what they actually buy

Roughly in order of value per unit of effort. The first two are worth more than the other five together, and both are architecture decisions rather than features.

1. Least privilege on tools

Start from an empty tool list and justify each addition. No general-purpose query tool, ever: expose get_order_status(order_id) rather than something taking SQL or an arbitrary filter. Split read from write and default to read. Bound each tool with a per-session call limit and, where money is involved, a ceiling the model cannot argue with. A tool needed for one user journey should not be in the list during the other nine. The largest single reduction in exposure available to you.

2. Tenant and row scoping enforced in the data layer

“Only answer questions about the current customer” in a system prompt is a request. A predicate the model cannot influence is a rule. Issue the database credential per request, scoped to the acting user, and let the database refuse rather than the model decline. The retrieval step needs the same treatment: filter by tenant before the similarity search, never after, and never from a string the model produced. Where that identity comes from is taken apart in build versus buy on authentication.

3. A confirmation step on the irreversible

Write down which actions cannot be undone and put a person in front of each. Two rules separate a control from a ritual. The confirmation must show the concrete action, the actual recipient, amount and record, never the model’s summary of its intent, because that summary is written by the component that was compromised. And the gate has to be rare: a dialog appearing fifty times a day is a button, not a decision. Approval fatigue is the standard failure mode, and the only answer to it is upstream, in mitigation one. Few tools, few gates, and few gates get read.

4. Treat the model’s output as untrusted input

Improper Output Handling is still on the OWASP ten in 2026 and it is the cheapest category to get right. Never pass model output to eval, a shell, or string-concatenated SQL. Render it as text rather than HTML. Allowlist link and image destinations, and switch off automatic loading of images at URLs the model wrote, the control that closes the EchoLeak-shaped hole above.

5. Structure the context so data looks like data

Cheap, partial, worth doing. Anthropic’s platform guidance is representative of where the vendors have landed: deliver third-party content inside tool result blocks rather than in the system prompt or a user turn, JSON-encode it so an attacker cannot close a tag and break out into an instruction context, label what it is and where it came from, and tell the model in the system prompt that tool content is data to report rather than commands to follow. That raises the cost of an attack. It does not stop a good one, and a vendor selling it as a solution is overselling.

6. Screening classifiers on the way in and out

A second, smaller model reads untrusted content or tool output and flags injection attempts before the main model acts. Real coverage against the attacks people run, with three caveats: it is probabilistic, it adds latency and a second line on the bill, and the screen is itself a model reading attacker text. Budget it as an extra call per tool result when you work out what the feature costs per user.

7. Adversarial evaluation, run as a regression suite

Write injection attempts into your fixtures the way you write SQL injection strings into an API test suite, and run them on every deploy. The public work to borrow structure from is AgentDojo, by Debenedetti and colleagues at NeurIPS 2024: 97 realistic tasks and 629 security test cases across email, e-banking and travel booking. Rerun it: a model upgrade, a new tool or a one-line prompt edit can each reopen something closed.

On what a good score looks like, a real number with its provenance attached. Anthropic reports a 1 percent attack success rate for Claude Opus 4.5 against their internal Best-of-N attacker given 100 attempts per environment, alongside the flat statement that no browser agent is immune. A vendor reporting on its own model, and still a genuinely good result. Then do the arithmetic: at ten thousand sessions a week, one percent is one hundred.

What buys close to nothing on its own is a line in the system prompt telling the model to ignore instructions found inside documents. Write it, because it is free and helps at the margin. Treating it as a control is the most common mistake here, and the one most often presented upward as though the problem were handled.

Where this argument runs against us

Most AI briefs we receive should not be agents. A workflow where your code picks the tool and the model only fills in a field has a far smaller attack surface: there is no tool selection for an injected instruction to capture. We argue that on determinism and cost grounds elsewhere, and security is the third leg of it. Which also means the cheaper thing we sell is frequently the safer one.

We are a software studio, not a security firm. We build with least privilege, scope tenancy in the database, gate irreversible actions and put adversarial cases in the test suite. We do not issue penetration test attestations, we do not run formal red team engagements, and we should not be the ones signing off a system carrying a regulatory obligation. In that case, retain a security practice for the review and keep a studio like ours for the build.

The design assumption is that the model will eventually be persuaded to try something. Not might. Will, given enough traffic and one motivated person. The OWASP project leads made the same point at the 2026 release: build the system so that when the model is fooled, and it will be, nothing important breaks. A design that holds only while the model is never fooled is not a design, and a vendor claiming their guardrails have solved this has told you enough.

What this costs

The mapping exercise, the tool inventory and the trust-boundary list are an afternoon with a whiteboard and cost nothing, which makes them the part with the best return. Money enters at design and build. Our published bands: an AI readiness assessment is $6,000 to $9,000 over two weeks, the right size for mapping the untrusted channels before anyone writes code. One production AI workflow runs $12,000 to $25,000, and a production LLM feature inside an existing product $30,000 to $45,000, both including the least privilege work and the adversarial suite rather than charging for them later. Ongoing AI advisory starts at $4,000 a month, for a team building this themselves and wanting the design reviewed. Those figures are estimates in US dollars, not quotations; the quote that follows scoping is fixed in writing before kickoff.

The recurring cost is easy to underestimate: screening adds a call per tool result, the adversarial suite is rerun on every model upgrade, and a confirmation gate spends human minutes daily. None of that shows in a build quote, and all of it shows in the second year.

The compressed answer

Prompt injection is unfixed and structural: instructions and data reach the model on one channel, so any untrusted text your system reads is potentially an instruction, including the text in your own user-writable database. The exposure is not what the model says. It is what it can call, what its credential can read, and which of its actions cannot be undone.

So: shrink the tool list, scope tenancy in the data layer rather than in the prompt, put a person in front of anything irreversible and keep those gates rare enough to be read, treat model output as untrusted input, structure and screen the context, and run adversarial cases on every deploy. Then assume all of it fails once, and check that the blast radius is survivable.

If you are scoping something with a model in it, send us the workflow: what text it reads, whose text that is, and what it may do afterwards. Those three answers are most of a threat model. If the honest conclusion is that you need a security firm rather than a studio, we will say so.