An LLM jailbreak is an input that gets a model to produce content its safety training is meant to block, without changing its weights. The main families are persona prompts like DAN, encoding tricks, many-shot prompts, multi-turn escalation and automated self-refinement, and studies report GPT-4 success rates from 87% (prompt libraries, on average) to 98% (automated attacks).
The term started as a loose label for DAN prompts traded on Reddit and Discord. It is now a research field with benchmarks, taxonomies and peer-reviewed success metrics, and its central finding repeats across papers: the gap between an aligned model and a safe deployment is measurable, and every production team has to plan around it.
Jailbreak or Prompt Injection?
A jailbreak is a type of prompt injection. OWASP LLM01:2025 treats prompt injection as any input that manipulates model behavior, and describes jailbreaking as the form where the attacker’s input makes the model disregard its safety protocols entirely. In a jailbreak the attacker is usually the user; in most other injections, the payload arrives inside third-party data.
The distinction is operational. A jailbreak unlocks disallowed content from inside an authorized session; a prompt injection usually weaponizes data flowing in from somewhere the user does not control. That determines what an engagement targets. Testing a chat assistant in isolation is mostly jailbreak hunting. Testing an agent that browses, ingests email or queries a vector store puts the indirect injection class formalized by Greshake et al. first: their paper planted instructions in data an application was likely to retrieve and used them to compromise real LLM-integrated systems, including Bing’s GPT-4-powered chat. Once retrieved content can carry instructions, the line between “jailbreak” and “remote takeover” collapses quickly. The direct vs. indirect prompt injection breakdown covers that side of the boundary.
Why Alignment Breaks
Safety alignment rests on two foundations, and both have structural gaps attackers work in. Reinforcement learning from human feedback trains a reward model on preference data and optimizes the LLM against it, so the reward model’s finite coverage means inputs far enough from the training distribution produce unexpected behavior. Supervised refusal fine-tuning is keyed to surface patterns (particular phrasings, topic areas, combinations), which is why rephrasing the same underlying request can evade a trained refusal. The model is not reasoning about intent. It is pattern-matching against a learned distribution, and that distribution has holes.
The January 2026 unified jailbreaking survey attributes susceptibility to three root causes: incomplete training data, since no dataset can cover everything a model will be asked; linguistic ambiguity, which lets a model misread user intent and produce inappropriate content; and generative uncertainty, where the generation process itself is subject to unresolved problems that surface as logical errors. No alignment technique closes all three at once. Sampling adds a practical wrinkle on top: the same prompt can be refused on one run and answered on the next, so a single clean run proves little.
Attack Taxonomy: Five Families Worth Knowing
Yi et al.’s 2024 survey splits jailbreak attacks by how transparent the target model is to the attacker: white-box or black-box. White-box attacks assume gradient access and optimize adversarial suffixes directly against the model; GCG and its variants live here. They need open weights, and their operational significance comes from transfer rather than direct use. That mechanism, along with attacker-LLM pipelines like PAIR and TAP, is covered separately in automated jailbreak attacks and the transfer problem. Black-box attacks assume only API access, which is the relevant category for production engagements.
Within the black-box space, five technique families recur. A 2025 evaluation of more than 1,400 adversarial prompts against GPT-4, Claude 2, Mistral 7B and Vicuna supplies comparable success rates for four of them.
Roleplay and persona assignment. The model is asked to assume a character (an AI with no restrictions, a creative writing partner, a historical figure) and the prohibited request is delivered in that character’s voice. Roleplay was the most effective category in the 1,400-prompt evaluation, at 89.6% attack success. It also dominates in-the-wild prompts, covered in the next section.
Encoding and obfuscation. Input filters keyed to prohibited keywords miss requests delivered in base64, Morse code, pig Latin, token substitutions or Unicode lookalikes; the model decodes and answers in plain text. Encoding tricks reached 76.2% in the same evaluation. The cheap extreme is FlipAttack, which disguises a harmful prompt by flipping its own text and then instructs the model to undo the flip. Its authors report about 98% attack success on GPT-4o within a single query, and about 98% average bypass against five guardrail models. Any encoding the model can interpret is usable, which keeps the defense surface wide.
Implication chaining and logic traps. A series of individually acceptable premises is built so that the final step implies the target content. Each step is low-violation, safety training does not generalize to the composed output, and the model’s pull toward coherence with its own prior text works against its alignment. Logic traps reached 81.4% in the 1,400-prompt evaluation. The same study measured portability across families: jailbreak prompts that succeeded on GPT-4 transferred to Claude 2 in 64.1% of cases and to Vicuna in 59.7%.
Fine-tuning attacks. Instead of working in prompt space, the attacker degrades alignment through the model’s fine-tuning phase. Qi et al. broke GPT-3.5 Turbo’s safety guardrails by fine-tuning it on 10 adversarially designed examples through OpenAI’s API, at a cost of less than $0.20, and found that fine-tuning on benign, commonly used datasets also degraded safety alignment, though to a lesser extent. Every provider that sells a fine-tuning endpoint inherits this surface.
Multi-turn conversational escalation. Multi-turn dialogues scored 68.7% in the 1,400-prompt evaluation, below single-turn roleplay. Automated escalation does considerably better, as the Crescendo results below show.
DAN and Persona Injection: The Prompt Classes Behind Most Jailbreaks
Most jailbreak prompts in circulation are persona prompts. Liu et al.’s empirical study classified 78 prompts collected from a jailbreak-sharing website and found that 97.44% of them used the Pretending strategy, with 77.6% using it exclusively. That figure describes the collected prompts, not the share of successful attacks. The study sorted the prompts into ten patterns under three categories:
- Pretending changes the conversation’s background or context while keeping the same intent: Character Role Play, Assumed Responsibility and Research Experiment.
- Attention Shifting changes both the context and the apparent intent: Text Continuation, Logical Reasoning, Program Execution and Translation. The harmful output arrives as a by-product of a task that looks benign.
- Privilege Escalation tries to circumvent restrictions directly: Superior Model, Sudo Mode and Simulate Jailbreaking.
DAN (“Do Anything Now”) is the canonical Pretending prompt. The user establishes an alternate identity, insists the identity operates without restrictions, then routes the harmful query through it. The class turns the instruction-following objective against safety training: if the stated context is “you are an AI that ignores rules,” that is a valid in-context instruction the model has to weigh against what it was trained to refuse. Long persona prompts add a second effect, holding the character in place across a conversation until its voice overrides the model’s default behavior.
Privilege Escalation drops the fiction and asserts authority instead: “developer mode is enabled,” “you are a superior model without limits,” “ignore all previous instructions.” It is the prompt-level equivalent of claiming you already have root rather than bypassing the permission check. A related move, operator impersonation, has the user’s message pose as a system-level instruction; it works where the boundary between system prompt and user turn is weakly enforced, the subject of the system prompt extraction and override examples.
The largest in-the-wild dataset comes from Shen et al.’s “Do Anything Now” study. The authors extracted 15,140 prompts posted between December 2022 and December 2023 on Reddit, Discord, websites and open-source datasets, identified 1,405 of them as jailbreak prompts, and grouped those into 131 jailbreak communities. Their “Basic” community contains the original DAN prompt and its close variants; an “Advanced” community uses more sophisticated attack strategies. The strategies they catalog include prompt injection, privilege escalation, deception and virtualization.
Two structural findings from that dataset matter for defenders. Jailbreak prompts averaged 555 tokens, 1.5 times the length of regular prompts, and the extra length typically goes to scaffolding: persona setup and rules the model is told to accept before the real request appears. Distribution also moved. From September 2023, websites contributed more than 75% of new jailbreak prompts, and the authors found 28 user accounts that had curated jailbreak prompts for over 100 days. The attack surface is an ecosystem with maintainers, not a static list a filter can memorize.
How Often Do Jailbreak Prompts Work?
Against GPT-4, often. Liu et al. report an average jailbreak success rate of 87.20% on GPT-4, and Shen et al. identified five prompts that reached a 0.95 attack success rate on both GPT-3.5 and GPT-4. The newer model resists better than its predecessor, but most attempts still succeed, and the scenario matters more than the model’s reputation.
Liu et al. ran 31,200 queries: 5 questions in each of 8 prohibited scenarios, through all 78 prompts, for 5 rounds, against both GPT-3.5-Turbo and GPT-4. Success rates fell in every scenario on the move to GPT-4, yet on average the upgraded model thwarted only 15.50% of jailbreak attempts. GPT-4 was notably strict on Harmful Content, where its jailbreak rate dropped to 45.2%. Across the study, the prompts consistently evaded restrictions in 40 use-case scenarios.
Shen et al. tested at larger scale: 107,250 samples across 13 forbidden scenarios against six models (ChatGPT on GPT-3.5, GPT-4, PaLM2, ChatGLM, Dolly and Vicuna). The earliest of their five most effective prompts had persisted online for over 240 days. Across all six models, Political Lobbying was the most vulnerable scenario at 0.855 ASR, followed by Legal Opinion at 0.794.
Two lessons follow for testers. First, patching the latest viral prompt does not retire the class; one of the best-performing prompts had been public for more than 240 days when measured. Second, categories with a less obvious harm model, like political and legal content, appear to be where refusal training is thinnest, and they deserve as much test coverage as chemical or weapons scenarios. Report results by category with a harm assessment attached: a 0.85 ASR on political lobbying is a different risk from a low rate on synthesis routes. The engagement methodology guide covers scoping and reporting at that level.
Three Research Attacks That Changed the Calculus
Many-Shot Jailbreaking
Anthropic’s many-shot jailbreaking research, published in the NeurIPS 2024 proceedings, describes an attack that only became practical once context windows grew from roughly 4,000 tokens at the start of 2023 to 1,000,000 tokens or more.
The attack is simple: fill the context with fake dialogues in which an AI assistant helpfully answers harmful questions, then append the real question. Anthropic tested up to 256 shots. Effectiveness follows a power law in the number of shots, so more shots means more compliance, and the attack worked against Anthropic’s own models and against models from other AI companies.
What makes many-shot operationally significant is that it requires zero optimization: no suffix search, no refinement loop. An attacker builds the fake dialogue once and replays it against any large-context target. Long context, marketed as a feature, created the attack class.
A dedicated breakdown of the technique (the scaling behaviour, adjacent attacks that share its shape, and how to scope it in a test plan) is in many-shot jailbreaking.
Mitigations are limited. Prompt-length monitoring can flag anomalously large inputs, but document summarization, code review and RAG pipelines produce large contexts too. Fine-tuning the model to refuse many-shot attempts merely delayed the jailbreak: it took more faux dialogues before harmful output appeared, but it still appeared. The strongest reported result is classifying and modifying the prompt before the model sees it, which in one case dropped attack success from 61% to 2%, at the cost of added latency and the risk of false positives on legitimate long-context work.
Crescendo: Multi-Turn Escalation
Crescendo, by Mark Russinovich, Ahmed Salem and Ronen Eldan and accepted at USENIX Security 2025, attacks the conversation rather than the single prompt. It opens with a general, benign question about the topic and escalates gradually by referencing the model’s own replies, each turn a step closer to the goal, until the model produces the prohibited content. The mechanism exploits a real property of autoregressive generation: a model that has already produced text on a topic weights that accumulated context heavily and continues in its direction. Crescendo builds the context deliberately, so compliance looks like continuation rather than a new decision.
The automated implementation, Crescendomation, beat other state-of-the-art jailbreak techniques on an AdvBench subset by 29–61% on GPT-4 and 49–71% on Gemini Pro. On average, most tasks were jailbroken in fewer than five turns, and the authors report that Crescendo effectively jailbroke every evaluated model (ChatGPT, Gemini Pro and Ultra, Llama-2 70B, Llama-3 70B Chat and Anthropic Chat) across the vast majority of tasks.
For a red team engagement, the procedural implication is direct: evaluation frameworks that test prompts in isolation measure the wrong surface. A deployment can pass every per-request safety check in a Crescendo sequence and still produce prohibited content a few turns in. The model is not bypassed; it is walked.
Defense requires conversation-level monitoring: semantic drift detection across a session, topic escalation fingerprinting, and tracking whether model-generated output is being used to prime subsequent requests. Per-request filtering is insufficient by design.
IRIS: The Model Jailbreaks Itself
IRIS (Iterative Refinement Induced Self-Jailbreak), by Govind Ramesh, Yao Dou and Wei Xu and published at EMNLP 2024, is one of the most compact automated attacks published. It needs only black-box access and uses a single model as both attacker and target.
The loop has two stages. In the first, when the target rejects an adversarial prompt, the attacker model is asked to explain why the attempt failed, then to modify the prompt; the refined prompt becomes the basis for the next iteration. Asking for an explanation first matters, because asking a well-aligned model to rewrite a rejected prompt directly tends to get refused. In the second stage, Rate+Enhance, the model scores the harmfulness of the output the refined prompt produced on a 1–5 scale and rewrites it to maximize that score, which catches responses that drifted into safe, educational content.
The reported results are 98% jailbreak success on GPT-4, 92% on GPT-4 Turbo and 94% on Llama-3.1-70B, in under 7 queries. The authors cap the loop at four iterations and note that more than 80% of the time only one or two are used.
What makes IRIS operationally relevant is what it does not need: no separate attack model, no gradient access, no curated dataset of working prompts. Input filters keyed to known-bad strings offer no coverage, because every run synthesizes fresh prompts. It also means the prompt-library success rates above are a floor: an attacker who iterates on structure does better, and IRIS automates the iteration.
The paper leaves defenses to future work. Two controls follow from the mechanism. Output-side classification evaluates the enhanced response no matter how the prompt was produced, and session-level detection of iterative probing (several structurally similar requests aimed at the same content category, interleaved with requests to explain refusals) gives a detection signal, at the price of false positives on legitimate iterative drafting.
The Multimodal Surface
Most jailbreak literature is text-only, but the attack surface follows the modalities the model accepts. The 2026 unified survey groups vision-language model jailbreaks into three classes with no text equivalent. Prompt-based image injection carries the request inside the image: FigStep, for example, rephrases a harmful question as a statement, renders it typographically as an image, and pairs it with a neutral text instruction to answer based on the image content. Combined prompt and image perturbations make small changes to both inputs to exploit the interaction between visual and text processing. Proxy model transfer generates adversarial inputs on an accessible surrogate model and carries them over to the target.
This matters for any deployment accepting image, audio or document input: RAG pipelines ingesting external files, agents with tool use, customer-facing assistants that process uploads. The surface scales with the number of input modalities, not with the size of the text interface, and once a model can act on what it reads, a successful jailbreak becomes a data-loss path, as agent tool-use exfiltration shows.
What Defense Actually Looks Like
Honest accounting from published literature: no single defensive measure stops all attack classes. Alignment training reduces success rates relative to unaligned models but does not eliminate jailbreaks. Constitutional AI and RLHF-based refusal training improve aggregate compliance but leave category-specific gaps. Keyword and pattern filters cover known bad prompts but are trivially bypassed by encoding, self-refining attacks that write new prompts every run, or multi-turn approaches that produce prohibited content without any single prohibited input.
Layered defense is the realistic posture:
- Input normalization to canonicalize encoding, strip zero-width characters, and decode common obfuscation before the model sees the prompt. This shrinks the obfuscation surface, and it is cheaper than classification.
- Input guardrails for known-pattern detection on top of normalized text, including the persona and privilege-escalation templates catalogued in the prompt studies.
- System prompt hardening with explicit refusal instructions, tested against known attack templates. Raises the baseline; replaces nothing.
- Conversation-level semantic monitoring for escalation, drift and iterative probing, especially for agentic systems where multi-turn interaction is the primary mode.
- Output filtering as a backstop, with classification against prohibited content categories independent of how the input was structured.
- Model diversity for high-stakes deployments, routing requests through a second model as a safety judge so that a model-specific jailbreak is not a single point of failure.
- Rate limiting and session analysis for detecting automated refinement loops and repeated probing.
The 2026 unified survey goes a step further and proposes defense principles by layer: variant-consistency and gradient-sensitivity detection at the perception layer, safety-aware decoding and output review at the generation layer, and adversarially augmented preference alignment at the parameter layer. They are worth understanding before committing to a production guardrail stack, because every control in the list above is keyed to attacks that have already been published. For how specific guardrail designs fail, see why prompt injection guardrails fail.
What to Add to Your Testing Playbook
The right corpus depends on what the target is. To rehearse the payload-crafting loop against a safe target before pointing it at a client system, the interactive AI Red Team Gym runs the same escalation pattern in the browser.
A chat assistant. Sweep the prompt-study taxonomy first: Pretending (DAN-style personas, fictional frames, character capture), Attention Shifting (text continuation, translation, program execution) and Privilege Escalation (sudo mode, superior model, operator impersonation), run against the full list of prohibited categories, political and legal content included. Then add encoded payloads (base64, character reversal, Unicode homoglyphs) and a long-context many-shot set. Score more than refusal versus compliance: partial leakage and off-by-one answers, where the model refuses the literal request and answers a paraphrase, are findings. Record success rate by category; the distribution is rarely uniform.
Multi-turn and self-refinement coverage. Build five-to-ten turn sequences that escalate from benign seed topics toward the target content, and check whether conversation-level monitoring fires before or after the escalation succeeds. Run an IRIS-style explain-and-refine loop against the highest-risk categories to see whether output filtering intercepts the enhanced result.
An agent or RAG system. Indirect injection through every ingestion path: crawled pages, uploaded documents, tool outputs, calendar invites, embedded image text. Test what happens when the model summarizes attacker-controlled content, and what happens when one tool’s output becomes another tool’s input. Confirm whether the system trusts retrieved text on the same footing as the user’s prompt. Most still do.
A model that claims to be defended. Run a transfer test with universal adversarial suffixes from the public sets, then a many-shot attack at the longest context the API will accept. Those two surface most of the easy wins before you spend budget on anything bespoke. Rerun whatever succeeds: if an attack works on one run and fails on the next, the defense is probabilistic rather than structural, and at production volume a probabilistic defense still fails. Record the exact model version too, since a jailbreak that fails on the latest checkpoint can still work on an older one that remains reachable through the API.
The DAN era trained a generation of practitioners to think jailbreaks are about clever wording. They are not. They exploit the gap between what a model is trained to refuse and what it is trained to do, and that gap appears everywhere the model trusts input it should not.
Teams building production LLM systems should assume the deployed model will be jailbroken by a determined attacker and build compensating controls at the application layer rather than trusting model-layer alignment as the sole defense, and keep the test corpus current with the literature rather than vendor announcements. To place these attacks in a wider frame, the adversarial ML attack taxonomy maps them onto the evasion, poisoning, extraction and inference structure compliance readers expect, and the AI Red Teaming Hub indexes the rest of the offensive AI security material on this site.
FAQ
what is the dan jailbreak prompt?
DAN, short for “Do Anything Now,” is a persona prompt that tells ChatGPT to act as an alternate AI with no restrictions, then routes harmful requests through that character. Shen et al. identify DAN as the original jailbreak prompt and place it, with its close variants, in the “Basic” community of their 1,405-prompt in-the-wild dataset.
what is the most effective llm jailbreak technique?
No single technique wins everywhere, but systematic attacks top the published numbers, though the benchmarks differ. IRIS reports 98% success on GPT-4 in under seven queries, and FlipAttack about 98% on GPT-4o in one query. Among prompt families, roleplay led a 1,400-prompt evaluation at 89.6%, ahead of logic traps at 81.4% and encoding at 76.2%.
can fine-tuning break an llm’s safety training?
Yes. Qi et al. broke GPT-3.5 Turbo’s safety guardrails by fine-tuning it on just 10 adversarially designed examples through OpenAI’s API, for less than $0.20. They also found that fine-tuning on ordinary, benign datasets degraded safety alignment to a lesser degree, so any team offering custom fine-tuning should rerun its jailbreak tests afterward.
how do you detect multi-turn jailbreaks?
Monitor the conversation, not the single request. Crescendo-style attacks pass per-message filters because each turn looks benign, so detection has to track semantic drift across a session, flag topics escalating toward prohibited categories, and notice when the model’s earlier output is being fed back to prime the next request. Output filtering remains the backstop when monitoring misses.
