What is an AI jailbreak?

Models are trained to say no to some requests: help with weapons, malware, abuse. A jailbreak is a prompt designed to get past that training, so the model does the thing anyway.

Nothing in the software is broken into. The attack is made entirely of words, aimed at the model's training and at the rules the app gave it.

How it relates to prompt injection

OWASP treats jailbreaking as one form of prompt injectionPrompt InjectionAn attack that hides instructions in text a model reads, such as a web page, an email or a file, so the model follows the attacker instead of you., and the 2 terms are often used for each other. A useful way to keep them apart is to ask who the attacker is.

• In a jailbreak, the user is the attacker. They want the model to break the rules set by its maker or by the app.
• In indirect prompt injection, the user is the victim. Someone else hid instructions in content the model reads for them.

The usual tricks

OWASP's list of example attacks shows the families. Most need no technical skill.

• Plain asking. Telling the model to ignore its rules, often dressed up as a game or a special mode.
• Disguise. Switching languages, or encoding the request (in Base64 or emojis, for example) so filters do not recognise it.
• Splitting. Breaking a request into harmless-looking pieces that only add up once the model combines them.
• Odd suffixes. A string of characters that looks like nonsense to a person but was found by search to tip the model over.

Anthropic describes Claude as resilient to these attacks by itself, and still tells builders to add their own layers on top.

How apps defend against it

Anthropic's guidance for builders layers several checks. A small, cheap model screens each message before the main model sees it. Inputs are checked against known attack patterns. The system promptSystem PromptInstructions the builder of an AI app gives the model before you type anything. It sets the role, the tone and the rules, and it outranks what users say. states the boundaries and tells the model how to refuse. Users who keep trying get warned, then limited or banned. Logs are watched, and the defences tightened. No single layer is enough on its own, which is why they are stacked.

Why it matters to you

If you build on a model, assume some users will try this, and decide in code, not only in the prompt, what your app may never do. If you use AI apps, know that a jailbroken answer has lost the checks that normally sit on it. It is not a hidden "true" answer. It is often a worse one.