Insecure Output Handling: LLM05:2025 Attacks and Defenses

Most write-ups about attacking language models stop at the moment the model says something it should not have said. That is the interesting half for a demo and the boring half for an attacker. The damage happens one step later, when whatever the model produced is handed to something that acts on it: a browser that renders it, a database that runs it, an interpreter that executes it, a shell that expands it.

That handoff is the vulnerability class OWASP originally filed as LLM02: Insecure Output Handling and renamed in the 2025 edition to LLM05: Improper Output Handling. Both names describe the same defect, so search results for either term are pointing at the same problem. This piece uses “insecure output handling” throughout because that is still the phrase most codebases and tickets use.

The one-sentence mental model

Model output is untrusted input.

That is the whole thing. Everything below follows from it. A language model is, from the application’s point of view, a remote service that returns attacker-influenceable text. It has no more claim to trust than a form field, a webhook body, or a scraped page. Applications get compromised because the output arrives wrapped in the vocabulary of an assistant, and assistants feel like part of the system rather than part of the internet.

The prompt injection compendium covers how an attacker gets influence over that text in the first place. Insecure output handling is what converts that influence into consequences. They are two different controls at two different layers, and fixing one does nothing for the other.

Why this is a separate finding from prompt injection

Teams collapse the two constantly, and the collapse is expensive, because it produces the wrong remediation.

• Prompt injection is a delivery problem. It answers: can an adversary shape what the model emits? Defenses are classifiers, spotlighting, trust separation, instruction hierarchies. None of them are reliable, which is why the bypass classes keep working.
• Insecure output handling is a sink problem. It answers: given that an adversary shaped the output, what does the application do with it? Defenses are output encoding, allowlists, parameterisation, sandboxing, and capability scoping.

The asymmetry matters. Injection defenses are probabilistic and degrade against new payloads. Sink defenses are deterministic and, once correct, stay correct. A system with unreliable injection detection and rigorous output handling has a bad day. A system with the reverse has an incident. When you triage findings on an LLM application, the sink side is where the fixable bugs are.

The five sinks that produce real findings

1. The browser: markdown rendering and stored XSS

Almost every chat interface renders model output as markdown. Most markdown renderers pass raw HTML through unless explicitly told not to. That is a direct path from model output to script execution in the user’s session, and if the conversation is saved and replayed to other users or to a support agent, it is stored XSS rather than reflected.

The variant that matters more, because it needs no script at all, is exfiltration through rendered images. Markdown image syntax makes the client fetch a URL the moment the message renders. If the model can be induced to build that URL with conversation contents in the query string, data leaves the session with zero clicks. The Microsoft 365 Copilot vulnerability tracked as CVE-2025-32711 (published as “EchoLeak”) followed exactly this shape: untrusted content entered the context, and the rendering layer completed the exfiltration. The agent tool-use exfiltration write-up covers the agent-side version of the same channel.

The mitigation is not “detect malicious markdown”. It is: disable raw HTML in the renderer, sanitise with an allowlist rather than a denylist, and set a Content Security Policy whose img-src and connect-src name the origins you actually serve. CSP is the control that turns a rendering bug into a blocked request in the browser console.

2. The database: text-to-SQL and generated queries

Natural-language-to-query features hand the model the query string. If that string is concatenated into a live connection, you have re-created SQL injection with a language model as the injection point. Graph and document stores are not exempt: CVE-2024-8309 covers Cypher injection through a LangChain graph question-answering chain, where the model wrote the query and the chain ran it.

Nothing about parameterised queries works here, because the model is generating the query, not the values. So the controls move up a level:

• Run generated queries as a role with read-only, column-scoped, row-scoped permissions. The database, not the prompt, enforces the boundary.
• Parse the generated statement and reject anything that is not a single SELECT. A parser, not a regex.
• Cap result size and execution time so a successful attack still cannot dump the table.

3. The interpreter: exec, eval, and generated code

The oldest LLM CVEs live here. CVE-2023-29374 describes a maths chain in LangChain that passed model-written Python to exec, so a prompt that talked the model into emitting arbitrary code got arbitrary code execution. CVE-2024-5565 describes the same defect one abstraction away: an analytics library asked the model to write plotting code, then executed the result, so prompt injection produced remote code execution in a product whose feature list said “ask your database a question”.

The pattern repeats because generated code is a genuinely useful product feature. If you ship it, the sink control is isolation, not inspection: a container with no network egress, no mounted credentials, a hard timeout, and a filesystem that does not survive the request. Scanning generated code for dangerous calls is a speed bump; a sandbox is a boundary.

4. The shell and the filesystem

Agents that build command strings, file paths, or URLs from model output inherit every classic injection bug: command injection through shell metacharacters, path traversal through ../, SSRF through a fetched URL that resolves to link-local metadata. The LLM security vulnerabilities survey shows how consistently these appear in the CVE record once agents get tools.

Do not build command strings. Pass argument arrays to the process directly, resolve paths against a fixed root and reject the result if it escapes, and put an egress allowlist in front of anything the agent fetches.

5. Structured output parsed into privileged fields

The subtlest sink. The model returns JSON, the application deserialises it and merges it into an object that governs behaviour: a role, a price, a tenant identifier, a tool name, a redirect target. This is mass assignment with a friendlier interface. Validate the parsed object against a strict schema, allowlist the fields you accept, and never let a model-produced value select the tool or the tenant.

Sink-to-control table

Sink	The classic bug it becomes	The control that actually holds
Markdown or HTML rendering	Stored and reflected XSS, zero-click data exfiltration	Raw HTML disabled, allowlist sanitiser, CSP on img-src and connect-src
Generated SQL, Cypher, or other query languages	Injection, unbounded reads	Read-only scoped role, statement parsing, result caps
exec, eval, generated scripts	Remote code execution	Network-isolated sandbox, no credentials, hard timeout
Shell commands, file paths, fetched URLs	Command injection, path traversal, SSRF	Argument arrays, root-anchored path resolution, egress allowlist
Parsed JSON merged into application state	Mass assignment, privilege escalation	Strict schema validation, field allowlist, no model-selected identifiers

Finding it in a codebase

Insecure output handling is unusually easy to review for, because the defect is always visible at a call site rather than distributed across a model’s behaviour. Trace every path that a completion string can take out of the model client and look for the moment it stops being a string:

• Renderers configured with raw HTML enabled, or a dangerouslySetInnerHTML-style call downstream of a completion.
• Any exec, eval, subprocess with shell=True, or deserialisation call whose argument descends from a model response.
• Query builders that take a completion as a full statement rather than as bound parameters.
• HTTP clients whose URL is assembled from model output with no allowlist.
• Response objects spread or merged into application state without a schema.

At runtime, the useful signals overlap with the ones in prompt injection detection: model output containing URLs the model was never given, output containing markup when the product never renders markup, and tool arguments that do not match the user’s request. Model output emitting an unexpected external hostname is one of the highest-signal, lowest-noise alerts available in an LLM application, and almost nobody wires it up.

Defense ordering that reflects real payoff

1. Encode at the sink, contextually. HTML-escape into HTML, parameterise into queries, argument-array into processes. This is the control that generalises across every payload that has not been invented yet.
2. Scope the capability the sink represents. A read-only database role and a network-isolated sandbox convert successful exploitation into a non-event.
3. Constrain the output format. Structured output with a strict schema removes most of the free-text surface. A model that can only return one of six enum values cannot return an image tag.
4. Content Security Policy. Cheap, deterministic, and specifically defeats the rendered-exfiltration channel that is otherwise invisible to every server-side control.
5. Human confirmation for irreversible actions. Sending mail, moving money, deleting records. Model output proposes; a person disposes.
6. Output classifiers last. They are useful for catching leaked secrets and policy violations. They are not a boundary, and a design that leans on them has no boundary.

The ordering is deliberate. Items one to four are properties of the code and hold regardless of how clever the payload is. Items five and six are compensating controls.

Where this sits in the wider attack surface

Insecure output handling is the reason indirect injection is worth an attacker’s time at all. A poisoned document that only changes what a chatbot says is a nuisance; the same document against an application that renders, queries, or executes the result is a breach. The direct versus indirect injection breakdown covers the delivery side, and the interactive Attack Technique Atlas maps the sink node against the rest of the offensive-AI graph, including which defenses each family answers to.

If you are prioritising work on an LLM application, this is the class with the best ratio of fix cost to risk reduction on the whole OWASP list. Detecting prompt injection is an open research problem. Not executing model output is a Tuesday.

---

→ This post is part of the AI Red Teaming Hub — the complete index of offensive AI security resources on aisec.blog.