What is red teaming in AI, and how do you red team your own LLM app?

Red teaming is attacking your own system before anyone else does. The name comes from military and security exercises, where a "red team" plays the enemy. For an AI system the attacks are words: a customer email that hides an instruction, a question built to pull out another user's data, a request phrased to slip past a refusal, a document planted where your search will find it.

Model red teaming and app red teaming

Model makers red team the model itself, looking for harmful answers. Anthropic's 2022 study red teamed models of 3 sizes and published 38,961 of the attacks people wrote. One finding: models trained with human feedback (RLHFReinforcement Learning from Human FeedbackA training step where people rank a model's answers, a reward model learns those preferences, and the language model is tuned to score well on it.) got harder to red team as they grew.

If you build on a model, your job is different. You red team your application: your system promptSystem PromptInstructions the builder of an AI app gives the model before you type anything. It sets the role, the tone and the rules, and it outranks what users say., your tools, your data and what your code does with the model's output. The attacker's goal is rarely a rude sentence. It is a refund nobody asked for, a customer's address sent to someone else, or your instructions printed in a reply. Prompt injectionPrompt InjectionAn attack that hides instructions in text a model reads, such as a web page, an email or a file, so the model follows the attacker instead of you. and jailbreaksJailbreakA prompt written to talk a model out of its safety rules, so it produces something it was trained or instructed to refuse. are the methods; the damage depends on what your app lets the model do.

How to red team an LLM app

1. List what could go wrong. OWASP's Top 10 for LLM applications is a good checklist: injection, leaked data, too much agency, a leaked system prompt, output passed on unchecked, runaway cost.
2. Write attacks as test cases. Each is an input plus what must not happen: no refund, no other customer's email address in the reply, no link to an outside site.
3. Grade the outcome, not the words. Check the database, the reply and the logs. A model that says "I can't do that" and then calls the tool anyway has failed.
4. Generate more. A model can write variations of each attack, the approach Perez and colleagues used to find thousands of failures no one had written by hand.
5. Run them on every change, like any eval setEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped., and add each real incident as a new case.

Red teaming finds failures; it does not prove there are none. That is why the defences that matter sit in code: a tool that only touches the sender's own orders holds whatever the model was talked into.