The full answer
A model's output is shaped by everything it read, including text an attacker controls through prompt injection, and it can also simply be wrong. So anything it produces must be handled like input from an unknown user: validated, encoded for where it goes, and never executed directly. OWASP lists the failure to do this as improper output handling (LLM05:2025), and it is how a fooled model becomes an exploited system.
What can go wrong
| Output goes to… | Without care | Attack |
|---|---|---|
| A web page | Inserted as HTML | Cross-site scripting: the model is tricked into writing a <script> tag or a malicious link |
| A database | Model-written SQL run directly | SQL injection, deleting or reading other tables |
| A shell | Command run as written | Remote code execution |
| An HTTP request | URL chosen by the model | Server-side request forgery: requests to internal services |
| A Markdown renderer | Images loaded from model-chosen URLs | Data sent out in an image URL's query string |
| Another system's API | Fields taken as given | Privilege escalation, wrong records changed |
What to do instead
- Encode for the destination. Escape HTML, use a Markdown renderer with raw HTML off, and restrict image and link domains.
- No model-written queries. Let the model pick parameters (a customer id, a date range) and run them through parameterised queries you wrote. If you must allow generated SQL (text-to-SQL), run it read-only, with a restricted role, row limits, timeouts and an allow-list of tables.
- No direct execution. Run generated code only in a sandbox with no secrets, no network or allow-listed network, and resource limits.
- Validate structure and values. Structured outputs make the shape reliable. Code must still check that values are allowed: ids exist and belong to this user, amounts are in range, URLs are on approved domains.
- Authorise as the user. Any action triggered by output runs with the user's permissions, not the service's, and irreversible actions need approval.
- Log it. Record outputs that triggered actions, for audit.
A concrete example
A support assistant writes summaries shown to agents in an internal dashboard. A customer's message contains a hidden instruction to include a markdown image whose URL embeds recent conversation text. If the dashboard renders images from any domain, the data leaves when the page loads. The fix is not a better prompt. It is restricting image domains and escaping output.
The principle to state: "The model is a user of my system, and not a trusted one. I apply the same zero-trust validation I'd apply to any input from the internet."
Last updated 16 October 2026
Sources