# What is red teaming in AI, and how do you red team your own LLM app?

Attacking your own AI system on purpose, the way a real attacker or a careless user would, to find where it fails before they do.

- Page: https://tecyfy.com/concepts/red-teaming
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: intermediate
- Also called: AI Red Teaming, LLM Red Teaming, Adversarial Testing

Red teaming is attacking your own system before anyone else does. The name comes from military and security exercises, where a "red team" plays the enemy. For an AI system the attacks are words: a customer email that hides an instruction, a question built to pull out another user's data, a request phrased to slip past a refusal, a document planted where your search will find it.

## Model red teaming and app red teaming

Model makers red team the model itself, looking for harmful answers. Anthropic's 2022 study red teamed models of 3 sizes and published 38,961 of the attacks people wrote. One finding: models trained with human feedback ([RLHF](https://tecyfy.com/concepts/reinforcement-learning-from-human-feedback)) got harder to red team as they grew.

If you build on a model, your job is different. You red team your application: your [system prompt](https://tecyfy.com/concepts/system-prompt), your tools, your data and what your code does with the model's output. The attacker's goal is rarely a rude sentence. It is a refund nobody asked for, a customer's address sent to someone else, or your instructions printed in a reply. [Prompt injection](https://tecyfy.com/concepts/prompt-injection) and [jailbreaks](https://tecyfy.com/concepts/jailbreak) are the methods; the damage depends on what your app lets the model do.

## How to red team an LLM app

1. **List what could go wrong.** OWASP's Top 10 for LLM applications is a good checklist: injection, leaked data, too much agency, a leaked system prompt, output passed on unchecked, runaway cost.
2. **Write attacks as test cases.** Each is an input plus what must not happen: no refund, no other customer's email address in the reply, no link to an outside site.
3. **Grade the outcome, not the words.** Check the database, the reply and the logs. A model that says "I can't do that" and then calls the tool anyway has failed.
4. **Generate more.** A model can write variations of each attack, the approach Perez and colleagues used to find thousands of failures no one had written by hand.
5. **Run them on every change**, like any [eval set](https://tecyfy.com/concepts/eval-set), and add each real incident as a new case.

Red teaming finds failures; it does not prove there are none. That is why the defences that matter sit in code: a tool that only touches the sender's own orders holds whatever the model was talked into.

## Related

- [Prompt Injection](https://tecyfy.com/concepts/prompt-injection): An attack that hides instructions in text a model reads, such as a web page, an email or a file, so the model follows the attacker instead of you.
- [Jailbreak](https://tecyfy.com/concepts/jailbreak): A prompt written to talk a model out of its safety rules, so it produces something it was trained or instructed to refuse.
- [Guardrail](https://tecyfy.com/concepts/guardrail): Checks placed around a model that stop or change what goes in and what comes out, such as a filter on user messages or an approval step before an agent acts.
- [Eval Set](https://tecyfy.com/concepts/eval-set): The list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped.
- [Alignment](https://tecyfy.com/concepts#alignment): Ensuring model objectives match human values and intent
- [Data Poisoning](https://tecyfy.com/concepts/data-poisoning): Planting text in the data a model learns from, or in the documents it searches, so it behaves the way an attacker wants.

## Sources

- [Red Teaming Language Models with Language Models](https://arxiv.org/abs/2202.03286)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://arxiv.org/abs/2209.07858)
- [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/)
- [Mitigate jailbreaks and prompt injections](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/mitigate-jailbreaks)