How do you red-team an LLM application before release?

red teaming means attacking your own system the way a motivated user would, before they do. Define what must never happen, build attacks against each of those outcomes (direct and indirect injection, jailbreaks, data extraction, abuse of tools, cost attacks), run them by hand and automatically, fix by changing the system's design rather than patching prompts, and keep every successful attack as a regression test.

1. Decide what you are protecting

List the harms specific to this system, for example:

• leaking another customer's data or the system prompt,
• taking an action without authority (a refund, an email, a deletion),
• producing content against the policy,
• giving dangerous advice in a regulated area,
• running up cost.

The OWASP Top 10 for LLM Applications is a good checklist to make sure nothing obvious is missing.

2. Map the attack surface

Every place text enters: the chat box, uploaded files, retrieved documents, web pages the agent reads, tool results, MCP servers, email it processes. Indirect inputs are where most real attacks come from, because users never see them.

3. Build attacks

• Direct: "ignore your instructions", role-play framing, other languages, encoded text (base64, homoglyphs), long multi-turn manipulation.
• Indirect: instructions hidden in a PDF (white text), in a web page, in a calendar invite, in a code comment.
• Data extraction: asking for other users' records, the system prompt, API keys.
• Tool abuse: getting the agent to call tools with attacker-chosen arguments (an email to an outside address, a URL that carries data out).
• Resource abuse: inputs that cause huge outputs or endless loops.

4. Run them

• Manual sessions by people who know the product. Creative attacks come from people.
• An automated suite: a library of known attacks, plus a model generating variations, graded by code checks (did the forbidden tool call happen?) and classifiers.
• Run against the whole system, with real tools in a sandbox, not just the bare model.

5. Fix by design

A prompt tweak that blocks one phrasing rarely blocks the next. Durable fixes reduce what success can achieve: remove a tool, narrow its permissions, add an approval step, validate arguments against allow-lists, separate untrusted content. Anthropic's guide on prompt leaks makes the same point: keep secrets out of prompts rather than relying on the model to guard them.

6. Keep it running

Every successful attack becomes a test. Rerun the suite when the model, prompt, tools or retrieval change. Monitor production for attack patterns, and give users a way to report bad behaviour.

What to report

Attack success rate per harm category, before and after fixes, and the residual risks you accept, with who accepted them.