Cross-Architecture Robustness of Prompt-Injection Defenses under Adaptive Attack Transfer

A research project about a question the agent-security literature keeps deferring: when a prompt-injection defense reports strong numbers in one experimental setup, does that number survive a change of agent architecture or a stronger attacker? Published defenses are usually evaluated in the scaffold their authors built, against the attack suite that was convenient, with the adaptive-attack methodology that was available at publication time. This project treats defense robustness as an empirical, measurable property and maps it: the same defenses, reimplemented across several open agent architectures, evaluated under both static and adaptive attack conditions, with the resulting transfer matrix reported honestly — including where defenses reorder, degrade, or stop mattering.

>Defensive security research on public benchmarks. Every experiment in this project runs inside published evaluation harnesses (AgentDojo, InjecAgent) against synthetic tasks in isolated environments. Nothing here touches real user systems, real credentials, or third-party services; the adversarial techniques studied are the ones already documented in the peer-reviewed literature, applied only to research fixtures.

Who Is This For?

• Security researchers who need to know whether a published defense is a property of the method or of the test rig
• MPhil/PhD students looking for a thesis with a clear artifact: a transfer matrix plus a defensible methodology for adaptive re-evaluation
• Agent platform engineers choosing defenses for production scaffolds that do not match any paper’s reference implementation
• Benchmark maintainers interested in how evaluation choices shape reported robustness

Research Problem

Prompt injection — attacker-controlled text that hijacks an LLM agent’s instructions — is the most-studied failure mode of tool-using agents, and a steady stream of defenses addresses it: structured input channels, self-checking pipelines, verification steps, detection classifiers. Each defense reports strong attack-success-rate reductions in its own evaluation. Two variables are almost never varied together: the agent architecture executing the defense (a monolithic tool-calling loop, a planner-executor split, or a multi-stage scaffold) and the attack strategy (the fixed benchmark cases versus an adaptive attacker that knows the defense). The research problem is precise: defense effectiveness measured in one architecture under one attack family may not transfer to others, and nobody has systematically measured how much of the reported robustness survives the transfer.

Why This Problem Matters

Deployment decisions are being made from single-rig numbers. A team running a planner-executor agent reads a defense paper evaluated on a monolithic scaffold and assumes comparable protection; an attacker does not assume anything — they adapt. The adaptive-attacks result (eight indirect-prompt-injection defenses bypassed with attack success above 50%) already showed that static evaluation flatters defenses; whether the magnitude of that flattering changes with architecture is unknown. If defense rankings reorder across scaffolds, the field’s implicit assumption — that robustness is a property of the defense alone — is wrong, and evaluation practice needs architecture-aware reporting. If rankings hold, the field gains a reusable negative result that saves everyone duplicate experiments. Either outcome is publishable and useful; the current literature cannot tell you which to expect.

Research Gap

What existing research has demonstrated

Recent studies have established the core building blocks. AgentDojo provides a standardized, extensible environment for evaluating prompt-injection attacks and defenses against tool-calling agents, and demonstrates that existing defenses do not fully solve the problem even in its own harness. The adaptive-attacks work then evaluated eight published defenses and bypassed all of them using attack strategies tuned to each defense’s assumptions. StruQ demonstrated that training-time structured queries substantially improve resistance with little utility cost; SelfDefend demonstrated that LLM self-checking pipelines can catch many injected instructions at inference time.

What remains insufficiently studied

Existing approaches focus primarily on a single agent scaffold per study, leaving architecture transfer insufficiently evaluated: no published work re-runs the same defense suite across heterogeneous agent architectures under a consistent adaptive-attack methodology, and no work reports which architectural properties (tool-loop shape, planner separation, observation formatting) predict residual robustness. The interaction is also unmeasured — adaptive attacks have been tuned per defense within one rig, but the joint variation of architecture × attack strategy × defense is absent from the literature. Defensible wording of the gap: recent studies have demonstrated defense fragility within their evaluation settings, but evaluation remains limited to single-scaffold rigs, so the cross-architecture generalization of both defenses and the adaptive-attack results themselves is open.

What Existing Research Shows

• AgentDojo (Debenedetti et al., NeurIPS 2024 D&B) — a dynamic environment of realistic agent tasks with programmable attack and defense hooks; showed that state-of-the-art models fail many tasks even without attacks and that existing defenses provide only partial protection in its harness.
• Adaptive attacks against indirect prompt injection (Zhan et al., Findings of NAACL 2025) — evaluated eight defenses and bypassed all of them with adaptive attacks, consistently achieving attack success above 50%, demonstrating that static-attack evaluation overstates robustness.
• StruQ (Chen et al., USENIX Security 2025) — training-time defense via structured queries that separate instructions from data; strong resistance with minimal utility loss in its evaluation setup.
• SelfDefend (Wang et al., USENIX Security 2025) — test-time defense where a checking LLM instance examines queries for injected intent; practical detection with measurable overhead.

What Remains Open

Three questions connect these findings to this project. First, do the adaptive-attack results themselves transfer — does a defense that collapses under adaptation in a monolithic scaffold also collapse in a planner-executor scaffold, or does architectural indirection absorb some attack pressure? Second, is defense ranking stable — is the ordering by attack success rate the same across architectures, or do defenses exchange places? Third, what explains residual robustness — can measurable architectural properties (how much untrusted content reaches the planning context, whether tool results are re-summarized, how commitments are sequenced) predict which defenses degrade least? None of these is answered by the existing papers, because none of them varies the architecture.

Research Question

> To what extent do prompt-injection defenses retain their effectiveness when both the attack strategy (static benchmark cases versus adaptive attacks) and the agent architecture (monolithic tool loop versus planner-executor versus re-summarizing scaffold) are varied — and which measurable architectural properties predict residual robustness?

Research Hypothesis

We hypothesize that defense rankings will reorder across architectures: prompt-hardening and self-check defenses will lose more effectiveness than isolation-style defenses when untrusted content passes through additional processing stages, because each transformation stage creates new injection paths that defenses evaluated on a single rig never exercised. We further hypothesize that adaptive-attack success will vary more across architectures than static-attack success, because adaptation exploits rig-specific assumptions that architectures expose differently.

Existing Research

• Debenedetti et al., “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” NeurIPS 2024 (Datasets and Benchmarks). Official page: https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html
• Zhan et al., “Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents,” Findings of NAACL 2025. Official page: https://aclanthology.org/2025.findings-naacl.395/
• Chen et al., “StruQ: Defending Against Prompt Injection with Structured Queries,” USENIX Security 2025. Official page: https://www.usenix.org/conference/usenixsecurity25/presentation/chen-sizhe
• Wang et al., “SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner,” USENIX Security 2025. Official page: https://www.usenix.org/conference/usenixsecurity25/presentation/wang-xunguang

Proposed Research Direction

The investigation proceeds in four movements. (1) Defense re-implementation. Select four to six published defenses spanning the design space — a training-time structured-input defense (StruQ-style, evaluated via its released patterns), a self-check defense (SelfDefend-style), a detection-classifier defense, and an isolation-style baseline — and reimplement them as architecture-independent wrappers whose behavior is documented per scaffold. (2) Architecture matrix. Run the same task suites on at least three open agent architectures built over the same underlying model: a direct tool-calling loop, a planner-executor split where the planner never sees raw tool output, and a scaffold that re-summarizes observations before they re-enter context. (3) Attack conditions. Evaluate under (a) the static benchmark attacks, and (b) adaptive attacks following the published adaptive methodology — attacker knows the defense, tunes prompts per defense per architecture, with a bounded adaptation budget so results are reproducible. (4) Analysis. Produce a transfer matrix (defense × architecture × attack condition) of attack success rates and utility deltas, then regress residual robustness against architectural features (context-path length, number of transformations of untrusted content, presence of re-summarization) to test whether robustness is predictable from architecture alone. Model choices should include one open-weights model family for cost control; the model is held constant across architectures so that architecture is the manipulated variable, not model identity.

MPhil Scope

A bounded, executable 6–12 month version: reimplement three defenses across two agent architectures on the AgentDojo benchmark, evaluate under static and adaptive attack conditions with a documented adaptation budget, and deliver (a) the transfer matrix, (b) a utility-cost analysis, and (c) a reproducible evaluation kit that others can extend to further architectures. This scope produces one focused contribution — the first controlled cross-architecture robustness measurement for a defense subset — without requiring novel attack invention or multi-model scaling.

PhD Extension

The doctoral program generalizes along three genuine axes. Adaptive generality: replace hand-tuned adaptive attacks with an automated adaptive-attack search that treats the defense-plus-architecture stack as the optimization target, testing whether the transfer conclusions hold against attack strategies no human tuned. Architecture independence: develop a characterization of defense-robustness predictors that transfers to architectures not in the training set of the analysis — a small benchmark of new scaffolds where predictions are tested blind. Certification direction: formalize what a “robustness claim” should mean for agent defenses (which variables must be held, varied, and reported for a claim to transfer), and propose an architecture-aware reporting standard, validated by auditing a wider defense corpus under it.

Dataset / Benchmark

• AgentDojo — a dynamic evaluation environment with realistic agent tasks, programmable injection attacks, and defense hooks. Official site: https://agentdojo.spylab.ai/ (code: https://github.com/ethz-spylab/agentdojo). Relevance: it is the de facto standard harness for exactly this evaluation class, its task/attack structure is extensible to new architectures, and both the adaptive-attacks methodology and several defenses were built against it, making results comparable to prior work. Suitable for MPhil experimentation: yes — public, scriptable, moderate compute per run.
• InjecAgent — a benchmark of indirect prompt injection test cases for tool-integrated agents covering user-directed harm and private-data exfiltration objectives. Official page: https://aclanthology.org/2024.findings-acl.624/ (code: https://github.com/uiuc-kang-lab/InjecAgent). Relevance: a second, differently-constructed suite to check that transfer conclusions are not AgentDojo-specific. Suitable for MPhil: yes, as the generalization test set.
• The adaptive-attack methodology itself is drawn from the published NAACL 2025 work; the adaptation budget and tuning protocol are part of this project’s documented contribution rather than a dataset.

Baseline Methods

• Undefended agent per architecture — the reference attack success rate that defenses are measured against.
• Static-attack evaluation of each defense — reproduces each paper’s own evaluation condition before adaptation is applied.
• StruQ-style structured queries — the strongest published training-time defense pattern, included as the reference point other wrappers must beat or match.
• SelfDefend-style self-checking — the strongest published inference-time checking pattern.
• Prompt-hardening baseline — a system-prompt instruction defense, included because it is the industry default and the literature suggests it is the weakest class.

Evaluation Metrics

• Attack success rate (ASR) under static and adaptive conditions, per defense × architecture cell.
• Utility delta — task success rate on benign tasks with the defense active versus undefended, per cell.
• Transfer delta — the change in a defense’s ASR between architectures, the project’s core novel measurement.
• Rank stability — Kendall-style rank correlation of defenses across architectures and attack conditions.
• Adaptation cost — number of attacker iterations needed to break a defense per architecture, as a robustness-depth proxy.
• False-positive rate on benign traffic per defense, so utility claims are attributable.

Experimental Design

Environment/dataset: AgentDojo as primary suite; InjecAgent as generalization suite; one open-weights model family held constant. Baseline: undefended + published defense configurations as shipped. Proposed method: architecture-independent defense wrappers with per-scaffold integration documented. Attack/test conditions: static benchmark attacks; adaptive attacks with bounded iteration budget (e.g., fixed candidate count per cell) and frozen attacker model. Control conditions: benign task runs per cell (utility), and a no-attack control to isolate defense-induced utility loss. Metrics: as listed above. Ablations: remove re-summarization from the indirection scaffold; replace planner separation with simple prompt segregation; vary adaptation budget to test sensitivity. Generalization tests: rerun the full matrix on InjecAgent; optionally one additional open scaffold if time allows.

Expected Research Contribution

A potential contribution takes the form of: (a) the first controlled cross-architecture robustness transfer matrix for prompt-injection defenses, (b) an empirical study establishing whether published rankings survive rig changes, and (c) a reusable evaluation framework (wrappers, adaptation protocol, analysis scripts) that subsequent work can extend. The project does not promise that defenses will fail or succeed — it promises the first trustworthy measurement of the transfer question, with either outcome contributing to evaluation practice.

Limitations

Reimplementation fidelity is the central threat: wrappers may not reproduce each paper’s exact integration, so per-cell absolute numbers carry uncertainty even though the comparative methodology is sound. The architecture set is small by necessity; findings are claims about the studied scaffolds, not about all possible agent designs. The adaptation budget bounds attacker strength — a longer-budget attacker might break defenses this study leaves standing, so results are lower bounds on attacker capability. Compute limits the model diversity; holding the model constant strengthens internal validity but weakens claims about model-dependence of transfer. Finally, AgentDojo/InjecAgent task realism is high but synthetic; production attack distributions may differ.

Research Ethics and Safety

All experiments run inside the published benchmark harnesses against synthetic tasks in isolated environments. The adaptive-attack methodology is applied only to research fixtures — never to third-party systems, real user data, or production agents — and attack artifacts are handled as research code with responsible-release practices (no weaponized payload collections beyond what the benchmark already publishes). The project’s purpose is defensive: measuring where defenses fail so that evaluation practice and deployed systems improve. No real-world exploitation, no unauthorized access, and no instructions for attacking live systems are produced at any stage.

Prerequisites

• Python and experience with LLM APIs or self-hosted inference
• Understanding of prompt injection and agent tool-calling architectures
• Experimental design and statistics (paired comparisons, rank correlation)
• Familiarity with at least one agent framework or the ability to read harness code
• Git; basic Linux sandboxing for reproducible runs

Suggested Research Timeline

• Phase 1 — Literature & methodology lock (weeks 1–4): defense selection, adaptation-budget protocol, success criteria.
• Phase 2 — Environment setup (weeks 5–7): AgentDojo operational; architecture variants built and verified equivalent on benign tasks.
• Phase 3 — Static evaluation (weeks 8–12): full defense × architecture matrix under static attacks.
• Phase 4 — Adaptive evaluation (weeks 13–19): bounded adaptive attacks per cell; iteration logs retained.
• Phase 5 — Generalization (weeks 20–23): InjecAgent rerun; ablations.
• Phase 6 — Analysis (weeks 24–27): transfer matrices, rank-stability, robustness regression.
• Phase 7 — Write-up (weeks 28–32): thesis/paper; release evaluation kit.

Related Ideas

• Reproducible Benchmark Harness for Desktop AI Agents — the measurement discipline this project applies to the security dimension of agent evaluation.
• Container Security Scanner for Docker Images — a sibling security-Idea illustrating how defensive tooling on the site treats evidence and reporting.
• Automated Web App Vulnerability Scanner — classical security evaluation practice that agent-security evaluation can borrow from.