LLM Red Team Engagement Checklist

A scoping-to-reporting checklist for LLM red team engagements. Use it whether you’re an internal team standing up an AI assessment program or an external consultancy adding LLM coverage. Pairs with the Prompt Injection Testing Checklist for tactical depth on the injection lane.

Phase 1 — Pre-engagement

Scope ambiguity kills LLM engagements faster than missing tooling. Lock the boundaries before you touch a model.

Authorization

• Written rules of engagement signed by both sides. Include model endpoints in scope, tools in scope, retrieval sources, and any sandbox vs production distinction.
• Third-party authorization for model providers (OpenAI, Anthropic, Google) — usage that looks like abuse can get test accounts banned without it.
• Cost cap in writing. LLM testing burns tokens. Cap the spend, document the rate, and assign an owner for overages.
• Data handling agreement — what can be exfiltrated to your tooling, what must stay on-prem, what must be deleted at end of engagement.

Scope

• System inventory: which models, which versions, what’s the application boundary, what tools are wired, what data sources feed retrieval.
• Out-of-scope items named explicitly (real customer data, payment endpoints, kill-switch infrastructure).
• Sandbox vs production distinction made clear. If only production is available, get explicit signoff on testing windows.
• Account provisioning: test accounts with appropriate roles. Note which roles you do and don’t have.

Threat model

• Attacker profile named: anonymous user, authenticated user, malicious insider, supply-chain compromise of training data.
• Crown jewels identified: which data, which actions, which downstream systems are the high-impact targets.
• Compensating controls mapped: WAF, prompt-firewall, rate limits, monitoring. Know what’s between you and the model.

Tooling

• Local test harness ready: garak, PyRIT, promptfoo, your team’s internal harness.
• Logging pipeline — capture every request/response with timestamps, model versions, and parameters.
• Burner browsers and accounts for any UI-based testing.
• Cost-tracking dashboard wired before the first request.

Phase 2 — Reconnaissance

Before exploiting anything, learn the system’s behavior.

• Model identification: ask the model what it is. Cross-check with response latency, token costs, and known fingerprints.
• System-prompt fingerprinting: probe for length, structure, behavioral constraints. Repetition tests, paraphrase tests, format-bleed tests.
• Tool inventory: enumerate which tools exist by triggering them. Look for tool errors that reveal names, signatures, or argument shapes.
• Retrieval surface: identify what triggers RAG retrieval, what corpus it pulls from, and whether you can poison it.
• Guardrail mapping: catalog refusal behavior. Which words trigger refusals? Which categories? What does a refused response look like vs an unfiltered one?
• Rate limits and abuse triggers: figure out the limits early so you don’t get throttled during the active phase.

Phase 3 — Execution

Work through each attack class systematically. Don’t lose findings in scratch buffers — log everything.

Attack classes to cover

• Direct prompt injection — see the Prompt Injection Testing Checklist.
• Indirect prompt injection via documents, retrieval, web fetches, and tool outputs.
• Jailbreak / safety bypass — characters, scenarios, encoding, multi-turn buildup.
• System-prompt extraction — direct, indirect, side-channel.
• Sensitive-data extraction — training data, customer data via context, PII via tool output.
• Tool-use abuse — forced calls, argument smuggling, exfil tools.
• Agent loop abuse — DoS, cost, infinite-loop, state corruption.
• Authentication and session boundary — does the model bleed across users/orgs?
• Output handling on the consumer — XSS via markdown, link-leak, image-beacon exfil in chat UI.
• Resource exhaustion — long-context attacks, cost burn, latency amplification.
• Model-version drift — does behavior change between model versions? Are there older/cheaper models accessible that lack guardrails?

Discipline

• Every finding has a minimum-viable PoC — the smallest reproduction that demonstrates the issue.
• Determinism check — run each successful attack 3+ times. LLM testing without reproducibility is noise.
• Negative space — note which attacks didn’t work and why. Helps the customer understand their defenses.
• Daily stand-up with the customer if engagement is >1 week. LLM findings are easy to misinterpret in writeups.

Phase 4 — Reporting

• Executive summary in plain English. The CEO needs to understand what’s at stake without learning what a token is.
• Technical findings, each with: title, severity, reproduction steps, evidence (logs, screenshots), business impact, recommended remediation.
• Severity model that the customer’s eng team can ingest — map to CVSS-adjacent and OWASP LLM Top 10 where possible.
• Remediation by layer: model-side, application-side, infrastructure-side, monitoring-side. Don’t dump everything on “the model team.”
• What you didn’t test — explicit out-of-scope items. Avoids false reassurance.
• Retest scope offered — what would constitute a fix verification pass.

Phase 5 — Post-engagement

• Data destruction per the data-handling agreement.
• Test accounts decommissioned at the customer site and the model provider.
• Findings disclosed to the model vendor where the issue is a model-side bug, not application-side misconfig.
• Knowledge capture — what worked, what didn’t, what to add to next engagement’s playbook.

Tools and references

• garak (NVIDIA) — LLM vulnerability scanner. Good for first-pass coverage.
• PyRIT (Microsoft) — Python framework for AI risk identification.
• promptfoo — eval and red-team framework, easy to script.
• OWASP LLM Top 10 — taxonomy customers expect to see in reports.
• MITRE ATLAS — adversary TTPs for AI/ML systems.

Engagement war stories, corrections, additions: editor@aisec.blog.