Attack Technique Atlas

Attacker-controlled text is interpreted by the LLM as instructions, overriding or hijacking the legitimate system prompt or user intent. The root family for most LLM application attacks.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant, RAG pipeline, Tool-using agent

The malicious instruction arrives in the user's own turn — the attacker is the user, talking directly to the model to override its system prompt.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant, Tool-using agent

The payload is embedded in third-party content the model retrieves or ingests (web pages, documents, emails, tool output). Because the model treats retrieved content as trusted context, attacker instructions execute without direct interaction.

Attacker capability

Black-box (query only), Training/RAG data access

Target type

RAG pipeline, Tool-using agent, Chatbot / assistant

Adversarial documents are inserted into the retrieval corpus so that, when retrieved as context, they steer answers or carry an indirect-injection payload.

Attacker capability

Training/RAG data access

Target type

RAG pipeline

Coaxing the model to reveal its hidden system prompt, developer instructions, or tool definitions — exposing business logic, guardrail wording, and downstream attack surface.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant, RAG pipeline

Eliciting content or behavior the model's safety training was meant to refuse, by manipulating framing, persona, or context rather than (necessarily) overriding a system prompt.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant, Vision / multimodal

The model is asked to adopt a persona or fictional frame ('DAN', 'evil twin', a movie character) for which refusal behavior is weaker, then steered to the disallowed content in-character.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant

A gradual multi-turn attack: benign questions establish context and incrementally steer the model toward disallowed output, exploiting the model's tendency to stay consistent with its own prior turns.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant

Dozens to hundreds of faux dialogue turns showing the assistant complying with harmful requests are stuffed into a long context window, biasing the model to continue the pattern.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant

The harmful request is hidden from input-side filters and from the model's refusal heuristics using base64, leetspeak, ciphers, low-resource languages, or token-splitting, then decoded by the model itself.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant

Greedy Coordinate Gradient optimizes a nonsensical token suffix that, appended to a harmful request, maximizes the probability of an affirmative response. Suffixes transfer across models.

Attacker capability

White-box (weights/logits)

Target type

Chatbot / assistant, Model API / endpoint

An attacker LLM iteratively rewrites prompts against a target, scoring responses to converge on a working jailbreak (PAIR, TAP, AutoDAN-Turbo, GPTFuzz). Scales discovery without human authors.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant, Model API / endpoint

Inputs perturbed by a small, often human-imperceptible amount that cause a model to misclassify or misbehave with high confidence.

Attacker capability

White-box (weights/logits), Black-box (query only)

Target type

Vision / multimodal, Model API / endpoint

Test-time perturbation crafted to flip a classifier's decision — e.g., evade a malware, spam, or content-moderation model — while preserving the input's real semantics.

Attacker capability

White-box (weights/logits), Black-box (query only)

Target type

Vision / multimodal, Model API / endpoint

Adversarial inputs crafted on a local surrogate model transfer to an unseen target, enabling black-box attacks without querying the victim heavily.

Attacker capability

Black-box (query only)

Target type

Model API / endpoint, Vision / multimodal

A localized, robust pattern (sticker, printout, eyewear) that forces a target class regardless of the rest of the scene, working under real-world capture conditions.

Attacker capability

White-box (weights/logits), Black-box (query only)

Target type

Vision / multimodal

Reconstructing a functional copy, or specific parameters, of a victim model by systematically querying it and training on the input/output pairs.

Attacker capability

Black-box (query only)

Target type

Model API / endpoint

Using a frontier model's outputs as soft labels to train a much cheaper student that approximates its behavior, circumventing licensing and capability investment.

Attacker capability

Black-box (query only)

Target type

Model API / endpoint

Recovering precise model artifacts — e.g., the embedding-projection layer or hidden dimension — from logit/logprob responses of a production API.

Attacker capability

Black-box (query only)

Target type

Model API / endpoint

Determining whether a specific record was in the model's training set, by exploiting the model's higher confidence on members. A privacy and compliance risk.

Attacker capability

Black-box (query only)

Target type

Model API / endpoint

Prompting a model to regurgitate verbatim memorized training data — PII, secrets, copyrighted text — sometimes via divergence or repetition attacks.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant, Model API / endpoint

Reconstructing representative or sensitive features of training inputs (e.g., a recognizable face for a class label) from model outputs or gradients.

Attacker capability

Black-box (query only), White-box (weights/logits)

Target type

Model API / endpoint

Corrupting training or fine-tuning data so the resulting model has degraded accuracy or attacker-chosen behavior.

Attacker capability

Training/RAG data access

Target type

Model API / endpoint, RAG pipeline

A hidden trigger is implanted during training so the model behaves normally until a specific trigger pattern appears, then produces the attacker's target output.

Attacker capability

Training/RAG data access, Supply-chain position

Target type

Model API / endpoint, Vision / multimodal

Injecting malicious preference pairs or annotations into RLHF/DPO data so the reward model — and the aligned policy — learns an attacker-chosen bias or backdoor.

Attacker capability

Training/RAG data access

Target type

Model API / endpoint

Compromising a model, dataset, package, or artifact before it reaches the victim's pipeline — abusing implicit trust in hubs, registries, and serialization formats.

Attacker capability

Supply-chain position

Target type

Model API / endpoint, Tool-using agent

Arbitrary-code serialization formats in model checkpoints (.bin, .ckpt) execute attacker code on load. Loading an untrusted checkpoint with an unsafe loader is remote code execution.

Attacker capability

Supply-chain position

Target type

Model API / endpoint

Exploiting a known vulnerability in an ML serving or pipeline component (model server, notebook, orchestration framework) to gain code execution or data access.

Attacker capability

Supply-chain position

Target type

Model API / endpoint

Publishing a malicious model or package under a name confusingly similar to a popular one so developers install the attacker's artifact by mistake.

Attacker capability

Supply-chain position

Target type

Model API / endpoint, Tool-using agent

Turning an LLM agent's own tools, memory, and autonomy against the user or system — typically bootstrapped by an injection that reaches the agent's planning loop.

Attacker capability

Black-box (query only)

Target type

Tool-using agent

Injected text instructs the agent to invoke a tool with attacker-chosen arguments (send email, run code, call an API), abusing the agent's action authority.

Attacker capability

Black-box (query only)

Target type

Tool-using agent

The model is induced to embed secrets in a rendered Markdown image/link URL or auto-fetched resource, leaking data to an attacker server with zero user clicks.

Attacker capability

Black-box (query only)

Target type

Tool-using agent, Chatbot / assistant

Planting persistent instructions in an agent's long-term memory or stored notes so the payload re-triggers on future, unrelated sessions.

Attacker capability

Black-box (query only)

Target type

Tool-using agent

An injected instruction makes the agent use its own elevated credentials/connectors on the attacker's behalf, reaching data or actions the attacker could not access directly.

Attacker capability

Black-box (query only)

Target type

Tool-using agent

Carrying an injection or jailbreak payload through a non-text modality (image, audio, document layout) that the model interprets as instructions.

Attacker capability

Black-box (query only)

Target type

Vision / multimodal

Instructions rendered as text inside an image (sometimes faint or off-color) are read by the vision-language model and followed as if user input.

Attacker capability

Black-box (query only)

Target type

Vision / multimodal

An optimized, near-imperceptible perturbation on an input image steers a vision-language model into producing disallowed text — adversarial examples meet jailbreaks.

Attacker capability

White-box (weights/logits)

Target type

Vision / multimodal

Forcing maximally expensive generations or runaway agent loops to exhaust the victim's token budget, rate limits, or downstream costs.

Attacker capability

Black-box (query only)

Target type

Model API / endpoint, Tool-using agent

Treating LLM output as trusted and rendering/executing it leads to classic injection (stored XSS, SQLi, SSRF) when an attacker can shape that output via prompt injection.

Attacker capability

Black-box (query only)

Target type

Chatbot / assistant, Tool-using agent

Detecting and stripping injection markers, role tokens, and obfuscation in untrusted input before it reaches the model. A first-pass signal, not a complete control.

Target type

Chatbot / assistant, RAG pipeline, Tool-using agent

A dedicated model (e.g., a fine-tuned classifier or LLM judge) that scores inputs/retrieved content for injection intent and blocks or quarantines them.

Target type

Chatbot / assistant, RAG pipeline, Tool-using agent

Delimiting, encoding, or datamarking untrusted content so the model can distinguish trusted instructions from retrieved data, reducing indirect-injection success.

Target type

RAG pipeline, Tool-using agent

Constraining which tools an agent may call, with argument schemas and human-in-the-loop confirmation for state-changing or sensitive actions.

Target type

Tool-using agent

Scanning model output for secrets, PII, and disallowed content — and neutralizing exfiltration vectors like auto-rendered image/link URLs — before it reaches the user or a sink.

Target type

Chatbot / assistant, RAG pipeline, Tool-using agent

Training on adversarial and red-team examples to harden the model against evasion and jailbreaks. Raises the cost of attacks but is not a guarantee.

Target type

Vision / multimodal, Model API / endpoint, Chatbot / assistant

Provenance, signing, safe serialization formats (safetensors), and scanning of models/datasets/packages before they enter the pipeline.

Target type

Model API / endpoint, Tool-using agent