Trajectory-Anomaly Pre-Commit Detection of Multi-Step Injection Attacks

A research project about catching agent attacks by how the agent behaves rather than by what any single message contains. Prompt-injection defenses today inspect messages — checking user input, scanning tool output, verifying a planned action against its trigger. Multi-step attacks are designed to defeat exactly that inspection: each individual step looks plausible, and the malicious intent only becomes visible in the sequence. This project investigates the alternative: learn what benign agent behavior looks like at the trajectory level — which tools in what order, with what argument shapes, at what pacing — and flag deviations before the agent commits the harmful action. The deliverable is a detection method evaluated on published multi-step attack suites, with an explicit early-detection measurement (how many steps before the damaging action the detector fires) and an explicit benign-feedback measurement (how much legitimate tool use the detector wrongly interrupts).

>Defensive research on published benchmarks. All experiments run against public attack suites (SIREN, InjecAgent, AgentDojo) in isolated environments with synthetic tasks. The project studies detection, not exploitation; no operational attack tooling is created, and nothing touches real agents, users, or systems.

Who Is This For?

• MPhil/PhD students in security or ML seeking a thesis with a clear method, public data, and a measurable defensive claim
• Agent-platform engineers building runtime monitoring who need evidence on trajectory-level versus message-level detection
• Security researchers in behavioral anomaly detection looking for a new domain with public benchmarks
• Trust & safety teams evaluating pre-commit intervention as a control point

Research Problem

Tool-using agents act in sequences: they plan, call tools, read results, and commit actions. A multi-step injection attack uses this structure — an early compromised step adjusts context subtly, a middle step retrieves attacker-influenced content, a late step commits the harm. Message-level defenses inspect each element in isolation and are documented to fail under adaptation; verification-style defenses (verify-before-commit) check actions against their triggering instruction but are pattern-dependent. The research problem: no published defense learns the normal trajectory distribution of an agent and uses deviations from it as the detection signal for multi-step injection campaigns before the harmful commit step executes. Behavioral anomaly detection is mature in network security; agents are a new, text-and-tool-shaped instance of the problem with public attack data available — but the transfer of the methodology has not been carried out or evaluated.

Why This Problem Matters

Pre-commit is the only intervention point where detection is cheap and harm is still avoidable. Post-hoc auditing tells you an attack succeeded; message-level filtering can be bypassed by design; verification can be steered. If trajectory-level signals carry real information about attack progress, a detector at the commit boundary could block the final harmful action while leaving the hundreds of benign tool calls untouched — the exact operating point production agent platforms need. The measurement also matters defensively in the other direction: if benign-trajectory diversity is so high that anomaly detection cannot separate attacks, that is a load-bearing negative result that redirects the field toward other control points, and it is currently unknown which answer is correct.

Threat Model and Assumptions

The attacker is assumed to control content that enters the agent’s context: tool outputs, retrieved documents, web pages, or inter-agent messages — the indirect-injection setting the cited benchmarks formalize. The attacker may chain multiple steps across turns, but does not control the agent’s system prompt, its tool implementations, or its memory store (state-persistent variants are the subject of a separate project on this site). The detector, by contrast, sees only what the runtime sees: the stream of tool calls, their arguments, their results’ metadata, and timing — not privileged model internals. Three assumptions bound the claims. First, benign behavior is learnable: the training corpus must be drawn from the same deployment distribution the detector will guard. Second, published multi-step attack suites approximate realistic campaign structure; a fully bespoke attacker may differ (addressed in the limitations). Third, the detector’s intervention point is the pre-commit boundary — it may warn, require confirmation, or block, but cannot rewrite the agent’s reasoning. These assumptions are stated up front so the evaluation measures the actual deployment question: given an in-distribution benign profile and published attack campaigns, how early and how reliably does behavior deviate?

Research Gap

What existing research has demonstrated

The verification paradigm is established: VIGIL, published with the SIREN benchmark of 959 tool-stream injection cases, demonstrated that verifying data and tool streams before commit defeats a range of tool-stream injection patterns. InjecAgent established that tool-integrated agents are broadly vulnerable to indirect injection across 1,054 test cases, including multi-turn settings. AgentDojo provided the standard harness connecting attacks, defenses, and utility measurement. ToolSafe demonstrated that step-level guardrails with feedback loops can improve tool-invocation safety without fully blocking the pipeline.

What remains insufficiently studied

Existing approaches focus primarily on inspecting message content or verifying actions against triggers, leaving behavioral sequence modeling insufficiently evaluated: no published work trains unsupervised trajectory-level models on benign agent behavior and evaluates them as pre-commit detectors for multi-step injection campaigns. Early-detection capability is also unmeasured — prior work reports whether an attack was blocked, not how many steps before the harmful action detection became possible, which is the operationally decisive quantity. The false-interruption cost on legitimate tool feedback — the practical objection to any behavioral gate — has not been quantified for trajectory detectors. Defensible statement of the gap: recent studies have demonstrated content-inspection and verification defenses, but trajectory-level behavioral detection with early-warning and false-block measurement remains open.

What Existing Research Shows

• VIGIL + SIREN (Lin et al., ACL 2026) — verify-before-commit paradigm defeats tool-stream injection in its evaluation; SIREN provides 959 cases with dynamic dependencies. Its detection is pattern/verification-oriented, not learned-from-benign-behavior.
• InjecAgent (Zhan et al., Findings of ACL 2024) — 1,054 test cases of indirect injection on tool-integrated agents; established baseline vulnerability levels that any detector must improve on.
• AgentDojo (Debenedetti et al., NeurIPS 2024 D&B) — the harness where attacks, defenses, and utility are measured jointly; provides benign trajectories as a side product of evaluation runs.
• ToolSafe (Mou et al., Findings of ACL 2026) — proactive step-level guardrail with feedback; the closest mechanism, but it guards individual steps rather than modeling whole-trajectory behavior.

What Remains Open

Three specifics remain unanswered. First, signal existence: do multi-step injection campaigns actually leave sequencing signatures (unusual tool order, argument-shape shifts, pacing anomalies) that distinguish them from benign trajectories, or is benign diversity too large for anomaly methods to separate? Second, timing: if signals exist, when do they first appear relative to the harmful commit — early enough to intervene? Third, cost: what benign-feedback block rate does a trajectory detector impose, and can confidence thresholds trade that rate against detection recall sensibly? SIREN’s cases and the benign runs from standard harnesses make all three measurable today; no published work has measured them.

Research Question

> Can unsupervised trajectory-level anomaly models, trained on benign agent execution traces, detect multi-step prompt-injection campaigns earlier than message-level detectors — and at what false-interruption cost on legitimate tool feedback?

Research Hypothesis

We hypothesize that injection trajectories exhibit detectable sequencing signatures before the harmful commit step — measurable as deviations in tool-order distributions, argument-shape statistics, and result-usage patterns — and that a trajectory detector trained on benign traces will achieve higher true-positive rates than message-level baselines on multi-step suites, while a confidence-thresholded operating point keeps the benign block rate within a small, measurable budget.

Existing Research

• Lin et al., “VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commit” (with the SIREN benchmark), ACL 2026. Preprint: https://arxiv.org/abs/2601.05755 (ACL 2026 long paper, DOI 10.18653/v1/2026.acl-long.443)
• Zhan et al., “InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents,” Findings of ACL 2024. Official page: https://aclanthology.org/2024.findings-acl.624/
• Debenedetti et al., “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” NeurIPS 2024 (D&B). Official page: https://proceedings.neurips.cc/paper_files/paper/2024/hash/97091a5177d8dc64b1da8bf3e1f6fb54-Abstract-Datasets_and_Benchmarks_Track.html
• Mou et al., “ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback,” Findings of ACL 2026. Official page: https://aclanthology.org/2026.findings-acl.1850/

Proposed Research Direction

The investigation compares four detector families under one protocol. (1) Data construction. Collect benign trajectories by running undefended agents over the benign splits of the chosen suites (AgentDojo benign tasks, InjecAgent’s clean cases); collect attack trajectories from SIREN and InjecAgent’s injected cases, replayed step by step. (2) Feature and model families. Evaluate: (a) a sequence model over tool-call event streams (order + argument embeddings); (b) a statistical baseline over n-gram tool-order and argument-shape features; (c) an LLM-judge that reads a window of the trajectory and flags deviation (the strong prompting baseline); (d) a hybrid that scores message-level anomaly plus trajectory-level deviation. (3) Pre-commit evaluation. At each step, the detector may flag; measure the step index of first flag relative to the harmful action’s index — the early-detection lead — and the eventual block outcome. (4) Cost calibration. Sweep decision thresholds to trace detection recall against benign block rate, reporting the Pareto frontier per detector family. The comparison strategy is fixed in advance: same folds, same thresholds grid, same benign corpus, so family differences are method differences, not evaluation artifacts. Model choices: one open-weights LLM for the agent under attack (held constant), one embedding model for argument representations; both documented for reproduction.

MPhil Scope

A bounded 6–12 month version: build the trajectory corpus from two public suites, implement two detector families (statistical baseline + one sequence model), evaluate against one message-level baseline and one VIGIL-style verification reference, and deliver the early-detection and false-block Pareto analysis on multi-step attack cases. One focused contribution: the first quantified answer to whether trajectory-level pre-commit detection works on published multi-step suites, with released code and corpus.

PhD Extension

The doctoral program extends along genuine axes. Adaptive evasion: model an attacker who knows the detector and optimizes trajectories to mimic benign statistics — measuring detector decay and designing adversarially-robust behavioral priors (e.g., per-tool calibrated envelopes, human-in-the-loop escalation on borderline trajectories). Architecture independence: test whether behavioral priors transfer across agent scaffolds and model families, or must be re-learned per stack, and develop few-shot calibration procedures for new stacks. Online deployment: streaming detection over live tool traffic with incremental learning, drift handling, and a formal treatment of the detection-latency/block-rate trade-off; extend to multi-agent settings where the trajectory spans several agents.

Dataset / Benchmark

• SIREN (with VIGIL) — 959 tool-stream injection cases simulating threats with dynamic dependencies; the primary multi-step attack source. Official code linked from the paper (https://arxiv.org/abs/2601.05755, ACL 2026). Relevance: purpose-built for the attack class this project targets, with step structure preserved. Suitable for MPhil: yes.
• InjecAgent — 1,054 test cases across 17 user tools and direct/indirect settings (https://aclanthology.org/2024.findings-acl.624/, code https://github.com/uiuc-kang-lab/InjecAgent). Relevance: injected versus clean cases give paired benign/attack trajectories for detector training and evaluation. Suitable for MPhil: yes.
• AgentDojo — benign trajectories from standard evaluation runs supply the benign corpus (https://agentdojo.spylab.ai/, https://github.com/ethz-spylab/agentdojo). Relevance: benign diversity from a standardized harness rather than self-generated runs. Suitable for MPhil: yes.
• Corpus assembly (joining these sources into a step-indexed trajectory format) is part of the project’s contribution; no new attacks are authored.

Baseline Methods

• Message-level detector baseline — a per-message injection classifier/prompt, the standard comparison point from the literature.
• VIGIL-style verify-before-commit reference — the published verification paradigm, run in its released form where feasible.
• LLM-judge trajectory reader — a strong prompting baseline that reads trajectory windows and flags attacks, representing “just ask a big model.”
• Random and always-flag controls — calibration references that make precision/recall numbers interpretable.

Evaluation Metrics

• Detection precision, recall, F1 over multi-step attack trajectories.
• Early-detection lead — mean/median steps between first flag and the harmful action (the project’s core novel metric).
• Benign block rate (false-positive rate) on legitimate tool feedback, per threshold.
• Pareto frontier — recall versus benign-block trade-off curves per detector family.
• Ablation-attributed signal contribution — detector performance with tool-order features removed, argument features removed, pacing removed.
• Cost per detection (tokens/compute), reported for the LLM-judge family where relevant.

Experimental Design

Environment/dataset: SIREN + InjecAgent attacks; AgentDojo + InjecAgent-clean benign corpus; one frozen agent model. Baseline: message-level detector, verification reference, LLM-judge. Proposed method: trajectory-level anomaly detector (two families), step-indexed evaluation. Attack/test conditions: full multi-step suites as published; cross-suite evaluation (train on one suite’s benign corpus, test on the other) as the generalization condition. Control conditions: benign-only runs (false-block measurement); attack-free replays (should never flag). Metrics: as above. Ablations: feature-family removal; threshold sweeps; benign-corpus size sensitivity. Generalization tests: cross-suite detection; one additional agent scaffold if time permits.

Expected Research Contribution

A potential contribution includes: a detection method (trajectory-level pre-commit anomaly detector with calibrated thresholds), an empirical study establishing whether behavioral sequencing signals exist and when they become detectable, and a released corpus/evaluation framework for pre-commit detection research (step-indexed trajectory format shared across suites). Results may support the method or constitute a well-evidenced negative result bounding where behavioral detection can help; either advances the defensive state of the art honestly.

Limitations

Benign-trajectory diversity in public harnesses is narrower than production traffic; detectors may look better here than in deployment, and the corpus is synthetic-benchmark rather than field data. Attack suites, while multi-step, contain human-designed patterns — a fully adaptive attacker might produce trajectories outside the evaluated distribution (the PhD extension addresses this; the MPhil result is bounded to published attack families). Feature choices (tool order, argument shape, pacing) may not capture all behavioral signal; unmodeled channels (timing, token lengths) remain untested. Compute constrains the number of detector families and scaffolds evaluated. Cross-suite generalization is measured on two suites, not the ecosystem.

Research Ethics and Safety

All experiments use published benchmark attacks replayed in isolated environments — no new attack tooling is created, no live systems are targeted, and no real user data is involved. Detector models and corpora are released for defensive research. The project explicitly avoids publishing anything that functions as operational attack guidance: attack material comes from peer-reviewed suites used as intended. The research objective is defensive measurement — establishing what protective capability behavioral detection can and cannot provide.

Prerequisites

• Python; PyTorch or equivalent for the sequence model
• ML fundamentals (unsupervised learning, evaluation methodology)
• Prompt injection and agent tool-calling concepts
• Statistics for threshold/Pareto analysis
• Git; comfort running public benchmark harnesses locally

Suggested Research Timeline

• Phase 1 — Literature & corpus design (weeks 1–3): suite selection, trajectory schema, metrics lock.
• Phase 2 — Corpus construction (weeks 4–7): benign + attack trajectories collected, step-indexed, validated.
• Phase 3 — Baselines (weeks 8–11): message-level, verification reference, LLM-judge implemented and measured.
• Phase 4 — Trajectory detectors (weeks 12–18): statistical + sequence-model families trained and tuned.
• Phase 5 — Pre-commit evaluation (weeks 19–23): early-detection and false-block measurement across thresholds.
• Phase 6 — Ablations & generalization (weeks 24–26): feature removal, cross-suite tests.
• Phase 7 — Analysis & write-up (weeks 27–30): Pareto analysis, thesis/paper, corpus release.

Related Ideas

• Reproducible Benchmark Harness for Desktop AI Agents — the trace-and-measure discipline underlying trajectory datasets.
• AI Test Case Generator from Source Code — sibling LLM-application research on the site with human-review boundaries.
• Privacy-First Offline Speech Transcription with Open Datasets — a complementary AI product Idea sharing the local-processing and evaluation rigor theme.