Medical LLM agents have an attack surface that most security teams treat as an academic concern. A new paper out of arxiv (2605.08257) makes that harder to sustain. The authors test four distinct attack classes against a medical decision-making agent, quantify the degradation each causes, and propose ARSM-Agent — a full-link six-stage defense framework with a weighted multi-objective training loss. Whether the defense numbers hold in production is a separate question; the threat model they lay out is worth reading regardless.
The Attack Surface
The paper tests four attack types that deserve attention individually.
Semantic perturbation. Inputs are modified in ways that preserve approximate meaning while shifting the token distribution outside the model’s confident decision region. In the medical context, this is analogous to paraphrasing a symptom description just enough to trigger model uncertainty, without changing the clinical meaning in a way a human reader would catch. This is the domain-general adversarial example problem applied to clinical text.
Prompt injection. The agent’s context includes externally retrieved medical evidence, patient history, or tool outputs. Injecting adversarial instructions into any of those sources can redirect the agent’s reasoning or override its safety constraints. The OWASP LLM Top 10 lists indirect prompt injection as a top risk precisely because medical and enterprise agents are both high-trust targets — they have access to sensitive data and their outputs carry weight. The paper tests this directly against the medical decision chain.
Drug-name confusion. This is the most domain-specific attack class and arguably the most practically dangerous. Drug names in real clinical systems are notoriously confusable — Hydroxyzine vs. Hydralazine, Clonidine vs. Klonopin — and the attack exploits that by introducing confusable drug names into prompts or retrieved evidence. A model that has not been explicitly hardened against this will sometimes treat a confused name as the intended drug, with obvious patient-safety implications. From a red-teaming perspective, this is a targeted semantic attack that exploits the model’s drug-name embedding space. It also has implications for any domain where terminology is dense and easily spoofed (financial instruments, network protocol names, CVE identifiers).
False-evidence injection. Retrieved documents or tool outputs that contain plausible but fabricated clinical evidence. Unlike prompt injection, which gives the model an instruction, false-evidence injection corrupts the factual grounding the model uses to reason. A RAG-backed medical agent that cannot distinguish authoritative evidence from adversarially fabricated content will hallucinate confident wrong answers. The paper frames this as a “knowledge consistency” problem, which is accurate — the agent needs to cross-check retrieved claims rather than accept them wholesale.
The ARSM-Agent Framework
The defense architecture is a six-stage pipeline the authors call “full-link”: input risk perception → medical evidence constraint → knowledge consistency verification → decision confidence reweighting → security output control → adversarial feedback update.
The framing matters. Rather than patching one layer (input filtering or output monitoring), the paper argues each stage in the decision chain is a distinct attack surface that needs its own control. That is the right architectural claim: an input filter that blocks semantic perturbations does nothing for a false-evidence injection that arrives through a legitimately retrieved document. Defense has to be distributed.
The more technically interesting piece is the training objective. The authors define a weighted joint loss:
- Decision accuracy loss — weight 0.3
- Adversarial robustness loss — weight 0.3
- Safety refusal loss — weight 0.2
- Knowledge consistency loss — weight 0.2
Treating adversarial robustness and decision accuracy as equal-weight objectives at training time is a deliberate tradeoff. Most production models optimize hard for task accuracy and treat safety as a post-hoc alignment step. Baking adversarial robustness in at the same training weight as accuracy means you are accepting some accuracy regression in exchange for a more robust decision boundary. The 0.2 weight on safety refusal is the model’s signal to decline genuinely dangerous outputs; the 0.2 on knowledge consistency trains it to flag retrieved evidence that conflicts internally or with established medical knowledge.
The paper compares ARSM-Agent against four baselines: LLM-Agent (no defense), Retrieval-Agent (RAG with standard retrieval), Filter-Agent (input/output filtering only), and Adv-Train-Agent (adversarial training without the multi-objective structure). ARSM-Agent outperforms all four across the attack types, though the abstract does not publish the margin numbers — those are in the full paper.
How This Changes the Engagement Playbook
For red teamers assessing medical AI systems, or any high-stakes agent deployment, this paper is useful as a threat model reference before the defensive claims get stress-tested.
The four attack classes generalize. Drug-name confusion is a specific instance of embedding-space semantic attack against domain jargon. Any agent operating in a terminology-dense domain — legal, financial, clinical, infosec — has an analogous attack surface. If you are building an assessment against a legal AI agent, the equivalent is case-name or statute confusion. For a financial agent, it might be ticker symbol manipulation or CUSIP spoofing in retrieved data.
False-evidence injection is the underappreciated one. Most prompt injection assessments focus on direct instruction injection into the user turn. False-evidence attacks target the agent’s knowledge source rather than its instruction layer, and they look like legitimate retrieval results. Testing this requires poisoning the retrieval corpus or simulating tool outputs, not just crafting adversarial user messages. Any RAG pipeline assessment should include this vector.
The ARSM pipeline is worth knowing because it will show up in vendor implementations as AI-powered medical tools harden. The six stages are natural detection points: if you can identify which stage handles evidence validation, you can probe its failure modes. A knowledge consistency verifier that relies on an LLM call is itself susceptible to prompt injection in the verification prompt.
The weighted loss design also implies something about the attack difficulty curve. A model trained with equal weight on accuracy and robustness is harder to degrade with semantic perturbations but potentially more brittle on inputs that fall outside the adversarial training distribution. If the adversarial training set did not include the specific drug-name pairs you construct during an engagement, the model may have no robustness there at all.
Skeptical Notes
A few things to verify before treating the ARSM results as production benchmarks.
The evaluation is conducted on a medical QA benchmark, not a deployed clinical system. Real medical agent pipelines include EMR API calls, multi-turn conversations with clinicians, and retrieval from heterogeneous document stores. The threat model in the paper applies; the defense numbers probably do not transfer directly.
The multi-objective loss weights (0.3/0.3/0.2/0.2) are presented as design choices without a hyperparameter search or ablation on the weights. It is not clear how sensitive the robustness/accuracy tradeoff is to those specific values. An adversary who knows the training objective could construct attacks that exploit the gap between the robustness and consistency loss terms.
The paper’s baseline comparisons are against relatively unsophisticated defenses. The real comparison class for a production deployment is against adversarially fine-tuned models with dedicated input classifiers and retrieval reranking. The margin over those baselines is not established.
None of this invalidates the threat model. It means the defense numbers are an upper bound on what you should expect in the field.
Related across the network
- AI Agents Are Rewriting the Threat Model, and Most Security Teams Aren’t Ready — techsentinel.news
- AI Content Moderation: How LLM Filters Work and Where They Break — guardml.io
→ This post is part of the AI Red Teaming Hub — the complete index of offensive AI security resources on aisec.blog.
