Security-Utility Pareto Measurement of Agent Defenses under Adaptive Attacks

A research project about the number defense papers don’t report: what an agent loses. Every prompt-injection defense publishes its attack-success-rate reduction; almost none publishes the complete curve of what that protection costs in task success, token overhead, and latency — and none reports that cost measured while an adaptive attacker presses on the defense. This project treats the security–utility trade-off as the primary object of study. For each defense in a representative set, the project sweeps its operating points (thresholds, checking intensity, strictness settings), measures both attack success and benign task success at every point, repeats the sweep under adaptive attack, and maps the resulting Pareto frontier: which defenses sit on the efficiency boundary, which are dominated once both axes are counted, and how the frontier itself moves when the attacker adapts.

>Defensive research on public benchmarks. All measurements run inside published evaluation harnesses (AgentDojo, InjecAgent) on synthetic tasks in isolated environments. The adaptive-attack methodology is the one documented in the peer-reviewed literature, applied only to research fixtures — never to real systems, users, or third-party services.

Who Is This For?

• MPhil students wanting an evaluation-science thesis with modest compute and a crisp deliverable
• Agent platform teams who must pick a defense and want the honest cost curve, not the marketing point estimate
• Security researchers studying evaluation methodology and the economics of defense deployment
• Benchmark maintainers interested in adding cost-normalized reporting to agent-safety suites

Research Problem

Defenses against prompt injection occupy a design space whose axes are security and utility. The literature evaluates the security axis at a single operating point per defense, usually against a static attack set, and treats utility as a secondary check (if it is measured at all). This creates two blind spots. First, a defense evaluated only at its strictest setting can look both safer and costlier — or both safer and cheaper — than it is across its reachable range, and single-point reporting hides which it is. Second, the adaptive-attacks literature demonstrated that static-attack evaluation flatters defenses; but the interaction is unmeasured: the utility cost a defense imposes may grow precisely where its adaptive-attack resistance collapses (for example, when strictness must be pushed to extremes to resist adaptation), which single-point, single-condition reporting cannot reveal. The research problem: the joint security–utility operating characteristic of agent defenses, under both static and adaptive attack conditions, is unmeasured — so the field cannot say which defenses are Pareto-efficient or how deployment decisions should be made.

Why This Problem Matters

Defense selection is currently an act of faith. A team choosing between a self-checking pipeline, a structured-input scheme, and a detection classifier has attack-success tables from different rigs, different models, and different utility accounting — no common curve, no common attacker, no way to see that Defense A blocks slightly fewer attacks while costing half the latency and none of the benign-task degradation. Pareto mapping fixes the decision: dominated defenses can be eliminated from consideration, efficient ones can be compared at matched utility budgets, and the adaptive-attack overlay shows which operating points evaporate under pressure. The result is directly usable by deployers and gives evaluation researchers a template for cost-normalized reporting that the field currently lacks.

Threat Model and Assumptions

The attack condition follows the published adaptive methodology: the attacker supplies indirect injection content through the channels the benchmarks provide (tool outputs, retrieved text) and, in the adaptive condition, knows which defense is active and may tune payloads against it within a fixed budget. The defender controls only the defense configuration — its operating parameters, not the underlying model’s weights (with the possible exception of training-time patterns applied per their published recipes). Two assumptions matter for interpretation. First, utility is measured on benchmark task distributions; production workloads with longer horizons may shift the cost axis, which is why the ablations reweight utility components rather than claim a single universal curve. Second, the adaptive attacker is budget-bounded so that every frontier point is reproducible; the measured contraction under adaptation is therefore a lower bound on real adaptive pressure. Within these bounds, the experiment answers a question neither defense vendors nor deployers can currently see: where each defense’s honest operating range lies, and what remains after an attacker presses on it.

Research Gap

What existing research has demonstrated

AgentDojo established the standard harness that measures attack success and task utility for agents under programmable attacks and defenses. The adaptive-attacks work (Findings of NAACL 2025) bypassed eight indirect-injection defenses with attack success above 50%, demonstrating that static evaluation overstates robustness. StruQ demonstrated a training-time structured-query defense with strong resistance and little measured utility cost in its own evaluation. ToolSafe demonstrated a guardrail-with-feedback design explicitly motivated by preserving task flow — evidence that the field recognizes utility loss as the binding constraint but has not systematically mapped it.

What remains insufficiently studied

Existing approaches focus primarily on security metrics at single operating points, leaving the joint operating characteristic insufficiently evaluated: no published work sweeps defense configurations to trace full security–utility curves, no work computes Pareto frontiers across defenses under a common harness and attacker budget, and no work measures how adaptive attacks reshape those frontiers. Cross-defense comparison also suffers from inconsistent utility accounting (different task sets, different overhead measurements), which this project removes by construction. Defensible statement: recent studies have demonstrated defense fragility under adaptation and individually promising cost profiles, but Pareto-level joint measurement across defenses and attack conditions remains open.

What Existing Research Shows

• AgentDojo (Debenedetti et al., NeurIPS 2024 D&B) — harness measuring utility and security jointly per run; provides the common rig this project requires.
• Adaptive attacks on indirect prompt injection (Zhan et al., Findings of NAACL 2025) — eight defenses bypassed with tuned attacks; establishes both the threat condition and the per-defense adaptation methodology.
• StruQ (Chen et al., USENIX Security 2025) — training-time defense with minimal reported utility loss; the strong “efficient frontier candidate” among published defenses.
• ToolSafe (Mou et al., Findings of ACL 2026) — step-level guardrail plus feedback loop; the design most explicitly engineered for utility preservation, and a natural frontier candidate.

What Remains Open

Three gaps connect directly to the project. First, curve existence: no defense has published its reachable security–utility curve, so even approximate dominance claims are impossible today. Second, cross-defense comparability: curves must be traced on one harness, one model, one task set to be comparable — which no study has done for more than two defenses. Third, adaptive reshaping: whether the frontier contracts, stretches, or reorders under adaptive attack is unknown — if strict operating points that resist adaptation also destroy utility, the practical frontier may be much thinner than static results suggest.

Research Question

> What are the security–utility Pareto frontiers of a representative set of prompt-injection defenses when operating points are swept on a common harness — and how do those frontiers change when a bounded adaptive attacker is introduced?

Research Hypothesis

We hypothesize that at least one widely cited defense is Pareto-dominated once its full operating range and benign-task cost are measured on a common rig, and that adaptive attack pressure will contract the frontier (fewer efficient points survive) while reordering some defense positions, with strictness-dependent defenses degrading earliest because their utility cost grows fastest as strictness rises.

Existing Research

• Debenedetti et al., “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” NeurIPS 2024 (D&B). Official page: https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html
• Zhan et al., “Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents,” Findings of NAACL 2025. Official page: https://aclanthology.org/2025.findings-naacl.395/
• Chen et al., “StruQ: Defending Against Prompt Injection with Structured Queries,” USENIX Security 2025. Official page: https://www.usenix.org/conference/usenixsecurity25/presentation/chen-sizhe
• Mou et al., “ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback,” Findings of ACL 2026. Official page: https://aclanthology.org/2026.findings-acl.1850/

Proposed Research Direction

The investigation is a controlled measurement campaign. (1) Defense selection. Four defenses spanning the design space: a structured-input/training-time pattern (StruQ-style), a self-check pipeline (SelfDefend-style), a detection classifier, and a guardrail-with-feedback design (ToolSafe-style). For each, identify its tunable operating parameters (strictness thresholds, check frequency, panel size, blocking mode versus flagging mode). (2) Sweep protocol. On AgentDojo with a frozen open-weights model, sweep each defense across at least five operating points; at each point measure attack success on the injected splits and task success, latency, and token overhead on the benign splits. (3) Adaptive overlay. Following the published adaptive methodology with a fixed attacker budget, repeat the sweep at each defense’s most promising points and measure how the security axis moves at unchanged utility cost. (4) Frontier analysis. Compute per-condition Pareto sets, dominance relations, and a Pareto-area metric; test the hypothesis by identifying dominated defenses and shifted points. The comparison strategy fixes harness, model, task split, and attacker budget in advance so all between-defense differences are attributable to the defenses themselves. Ablations vary the utility metric weighting (task success versus latency versus cost) to check that conclusions are not artifacts of one weighting choice.

MPhil Scope

A bounded 6–12 month version: three defenses on one harness (AgentDojo), five operating points each, static plus one adaptive-attack configuration, with the full Pareto analysis and released sweep infrastructure. This delivers the first common-rig frontier measurement for a defense subset — one focused, defensible contribution — while keeping compute within a single-GPU-plus-API budget by using an open-weights agent model and bounded sweep grids.

PhD Extension

The doctoral program deepens the measurement into theory and practice. Adaptive co-optimization: let the attacker and the defense-selection process interact over rounds — measuring frontier evolution under an adaptive adversary with unbounded retuning, and designing defense portfolios that present convex frontiers to attackers. Cost modeling: develop a principled cost model (risk reduction per utility dollar/token) and derive deployment guidelines as an optimization problem; validate on tasks and models outside the MPhil rig. Certification: formalize what a defensible defense claim must include (frontier, conditions, attacker budget) and audit a wide defense corpus against that standard, building the field’s first cost-normalized defense registry.

Dataset / Benchmark

• AgentDojo — primary harness: programmable attacks and defenses, joint utility/security measurement, public and scriptable (https://agentdojo.spylab.ai/, https://github.com/ethz-spylab/agentdojo). Relevance: the only common rig that already measures both axes per run, making sweep economics tractable. Suitable for MPhil: yes.
• InjecAgent — generalization suite for the frontier conclusions (1,054 indirect-injection cases; https://aclanthology.org/2024.findings-acl.624/, https://github.com/uiuc-kang-lab/InjecAgent). Relevance: checks that frontier shape is not AgentDojo-specific. Suitable for MPhil: yes, as the final replication pass.
• Adaptive-attack protocol from the NAACL 2025 Findings paper supplies the attacker methodology and budget convention; the sweep grids and cost accounting are this project’s documented protocol.

Baseline Methods

• Undefended agent — the origin point (zero protection, full utility) anchoring every frontier.
• Each defense at its published default setting — reproducing the single points the literature reports today.
• Static-only frontier per defense — the intermediate result that the adaptive overlay then amends.
• Prompt-hardening baseline — the industry default, expected to be dominated but included for completeness.

Evaluation Metrics

• Attack success rate per defense × operating point × attack condition (static/adaptive).
• Benign task success rate per operating point.
• Overhead metrics: added latency per task and token cost per task, normalized against undefended runs.
• Pareto dominance counts and Pareto-area per defense per condition.
• Frontier shift delta — the movement of each defense’s curve between static and adaptive conditions.
• Rank stability — whether defense ordering by Pareto-area changes across utility-metric weightings.

Experimental Design

Environment/dataset: AgentDojo primary; InjecAgent replication; frozen open-weights agent model. Baseline: undefended + published-default points. Proposed method: full sweep grids per defense with unified cost accounting. Attack/test conditions: static suite attacks; adaptive attacks with fixed budget at selected operating points. Control conditions: attack-free benign runs per point (utility); repeated-seed runs to bound run-to-run variance before frontiers are computed. Metrics: as above. Ablations: utility-metric reweighting; sweep-grid density sensitivity; attacker-budget sensitivity at the adaptive stage. Generalization tests: InjecAgent frontier replication; optional second agent model to test model-dependence of frontier shape.

Expected Research Contribution

A potential contribution includes: the first cross-defense security–utility Pareto measurement on a common harness under static and adaptive conditions; an empirical study identifying dominated defenses and frontier contraction under adaptation; and a released evaluation framework (sweep infrastructure, cost-accounting protocol, frontier analysis toolkit) that turns single-point defense reporting into curve reporting. The project does not presuppose which defenses dominate — the measurement, whichever way it lands, is the contribution.

Limitations

Sweep breadth trades against compute: grids are bounded, so curves are samples rather than continuums, and narrow operating-point spacing near the frontier is not guaranteed. Reimplementation fidelity applies here as in all defense-reproduction work; wrappers are documented but absolute numbers carry implementation uncertainty (comparative claims are the robust output). The attacker is bounded by budget — frontier contraction is a lower bound on adaptive pressure. Utility is measured on benchmark tasks, whose overhead structure (short tasks, few tool calls) may understate production costs on long workflows. Model-dependence is only partially explored within MPhil scope.

Research Ethics and Safety

All experiments run inside public benchmark harnesses on synthetic tasks in isolated environments. Adaptive-attack methodology is applied strictly per the published protocol to research fixtures; no real systems, user data, or third-party services are involved, and no operational attack tooling beyond the published suites is produced. The purpose is defensive economics — enabling deployers to choose protections honestly and researchers to report costs alongside security claims.

Prerequisites

• Python; comfort running LLM inference (API or self-hosted)
• Basic experimental design and statistics (variance estimation, curve comparison)
• Understanding of prompt injection and agent tool-calling
• Familiarity with at least one agent evaluation harness (or willingness to learn AgentDojo)
• Git; spreadsheet/plotting literacy for frontier analysis

Suggested Research Timeline

• Phase 1 — Literature & protocol lock (weeks 1–3): defense set, sweep grids, cost accounting frozen.
• Phase 2 — Infrastructure (weeks 4–6): AgentDojo operational; sweep runner implemented; variance calibration.
• Phase 3 — Static sweeps (weeks 7–13): full grids for all defenses; interim frontiers.
• Phase 4 — Adaptive overlay (weeks 14–19): bounded adaptation at selected points; amended frontiers.
• Phase 5 — Replication (weeks 20–22): InjecAgent pass; ablations.
• Phase 6 — Analysis (weeks 23–26): dominance analysis, hypothesis tests, frontier visualizations.
• Phase 7 — Write-up (weeks 27–30): thesis/paper; release sweep kit.

Related Ideas

• Reproducible Benchmark Harness for Desktop AI Agents — the common-rig measurement philosophy this project extends from execution scoring to defense economics.
• AI Test Case Generator from Source Code — sibling LLM-application research with evaluation-rigor framing.
• Automated Web App Vulnerability Scanner — classical security-evaluation practice whose reporting norms this project modernizes for agents.