Many-Shot Jailbreaking: How Long Context Breaks Alignment

Most jailbreak techniques are shaped like a puzzle. Find the phrasing, the persona, the encoding, the suffix that slips past whatever the model was trained to refuse. They are creative, they are brittle, and they get patched.

Many-shot jailbreaking is not shaped like a puzzle. It is shaped like a dial. The attacker writes a long list of fabricated exchanges in which an assistant cheerfully answers requests it should refuse, pastes the whole list in as context, and then asks the real question at the end. Nothing is obfuscated. Nothing is disguised. The only variable is how many fabricated exchanges are in front of the question, and the published results say that turning that dial up keeps working.

Anthropic documented the technique in 2024, reporting that attack effectiveness scaled as a power law with the number of in-context demonstrations and held across a range of harm categories. The accompanying paper (Anil et al., NeurIPS 2024) reports the same behaviour on models from several different developers — Claude 2.0, GPT-3.5 and GPT-4, Llama 2 70B and Mistral 7B. Anthropic states that it briefed other AI developers in advance of publication, which is the appropriate handling for a finding that is not specific to one vendor’s safety training.

Why the attack exists at all

The uncomfortable part is that many-shot jailbreaking is not exploiting a bug. It is exploiting the feature that makes these models useful.

In-context learning is the property that lets a model infer a task from examples in the prompt without any weight updates. Show it three translations and it translates. Show it ten JSON objects in a schema and it emits an eleventh. That capability improves predictably with more examples, and the improvement follows the same kind of smooth scaling curve everywhere it has been measured.

Safety training is a different kind of thing. It is a learned disposition, applied at the level of the response, competing with everything else the context is telling the model to do. When the context contains two hundred consistent demonstrations of an assistant behaving one way, the model faces a conflict between a pattern it has just been shown at length and a preference it acquired during training. The pattern wins often enough to matter, and it wins more often as the pattern gets longer.

That is why the fix is hard. You cannot remove in-context learning. It is the product.

The context window is the attack surface

The technique became practical because context windows grew. A model with a few thousand tokens of context cannot hold hundreds of fabricated exchanges plus the payload. A model with a few hundred thousand tokens can hold as many as the attacker cares to write, and long context is a headline feature that no vendor is going to walk back.

This is worth stating plainly, because it inverts the usual assumption about capability and safety. Every context-window increase is a capability release and a jailbreak-surface release in the same shipment. The same is true of the adjacent features: prompt caching makes long adversarial prefixes cheap to reuse, and multimodal inputs let the demonstrations arrive as images or transcripts rather than as text a filter would see.

The LLM jailbreak taxonomy treats this family as its own branch for that reason. It does not belong with role-play framing or encoding tricks. Those attacks work because of what they say. This one works because of how much of it there is.

The family it belongs to: attacks that scale rather than trick

Many-shot jailbreaking is the clearest member of a family that shares one property — the attacker’s main input is volume, not cleverness.

Crescendo. A multi-turn escalation described by Russinovich, Salem and Eldan, in which every turn is individually benign and each one leans on the model’s own previous answer to move a step further. The model is not being tricked into ignoring its instructions; it is being invited to stay consistent with itself, and consistency carries it somewhere it would have refused to go in one jump. The published work includes an automated version, which is what makes it operationally interesting rather than a party trick. Per-turn content filters see nothing, because the payload only exists in the trajectory.

Payload splitting. The disallowed content never appears in one place. It is broken into fragments distributed across the context — variables, list items, separate turns — and the model is asked to assemble them at the end. Input classifiers score fragments, and fragments score clean. This is the same evasion logic as many-shot, applied to the payload instead of the demonstrations.

Prefill. Where an interface allows the caller to supply the opening of the assistant’s turn, the attacker writes the first words of a compliant answer and lets the model continue from a position where refusal is off the conversational path. Cheap, mechanical, and unavailable in chat products but frequently available in raw APIs.

Long-context distraction. Bury the request deep inside a large volume of unrelated material, so that the instruction block is a small fraction of what the model is attending to. This is the same lever as many-shot with the demonstrations replaced by filler, and it is why long-context injection attacks against retrieval systems are worth running at all — see the FlashRT coverage for the compute-cost side of that.

The common thread is that all four defeat the standard defensive posture in the same way. A guardrail that scores one input at a time cannot see an attack whose only distinguishing feature is its shape across many inputs. That is the same structural failure the bypass classes piece describes for injection.

Why it transfers between models

Techniques that depend on a specific tokenizer or a specific gradient — adversarial suffixes, for example — transfer unevenly, and the transfer problem is the main practical limit on automated attack pipelines.

Many-shot has no such dependency. It uses no model internals, no logits, no white-box access, and no artifact of one vendor’s training data. It needs a long context window and instruction-following, which is to say it needs the two properties every current frontier model is sold on. An attack whose only prerequisites are the product’s headline features transfers by construction, and that is the finding that should shape how a red team budgets its time.

What actually blunts it

The published mitigation results are the most useful part of the research, mostly because of what they rule out.

Refusal training helps less than it looks. Fine-tuning against the attack raises the number of demonstrations required rather than removing the vulnerability. The curve shifts right; it does not flatten. That is a rate limit, not a fix, and it degrades as context windows grow.

Shortening context works and is unacceptable. Capping the window defeats the attack and destroys the feature. Nobody is shipping it.

Classifying and rewriting the prompt before it reaches the model works best. Anthropic reported that a prompt-based classification and modification step cut attack success from 61% to 2% in one evaluation. The important detail is where the control sits: it is a preprocessing layer over the whole context, not a filter on the last user turn. Anything that inspects only the final message is looking at the one part of a many-shot attack that is completely innocuous.

For an application team, that translates into a short list:

1. Score the whole context, not the last turn. Structural features are the signal — repeated dialogue-like formatting, an unusual ratio of assistant-voiced text in the user-supplied region, hundreds of near-identical exchange blocks.
2. Treat user-supplied conversation history as untrusted. Applications that let a client post an entire message array are handing the attacker the demonstrations for free. Server-side session state removes the primitive entirely.
3. Watch refusal rates as a time series. A session whose refusal rate falls monotonically as context length grows is the observable signature of both many-shot and Crescendo. This is the cheapest detection available and it belongs with the rest of the detection signals.
4. Score across turns, not within them. Escalation is only visible in a trajectory. Per-turn guards are structurally blind to it.
5. Do not let content policy be the last boundary. If a jailbroken response can only produce text, the impact is bounded by what that text is allowed to touch — which is a question about output handling and capability scoping, not about alignment.

What this means for red-team scoping

Many-shot deserves a standing slot in an LLM test plan, and it should be run as a scaling measurement rather than a pass/fail probe. The useful deliverable is not “the model refused”. It is a curve: attack success against number of demonstrations, per harm category, at the context length your product actually ships. That curve tells a product owner what their context-window setting costs them, which is a decision they can act on. A binary result tells them nothing, because every model refuses at ten shots and the attacker is not stopping at ten.

Pair it with a Crescendo-style multi-turn run against the same targets, since the two attacks stress the same missing control from different directions, and record the shots-to-first-compliance number so the next release can be compared against it. The jailbreak taxonomy and the interactive Attack Technique Atlas both map where these sit relative to the rest of the offensive surface, and Jailbreaks FYI tracks the technique-level catalogue as it moves.

The strategic read is uncomfortable and worth saying without hedging: this class of attack gets stronger with every capability release, and the only defenses that have held so far live outside the model. Plan accordingly.

---

→ This post is part of the AI Red Teaming Hub — the complete index of offensive AI security resources on aisec.blog.