Post-Hoc Recovery Evaluation: Measuring Agent Recovery After Unsafe Tool Execution

A research project about the moment after the mistake. Agent benchmarks overwhelmingly measure whether an agent avoids unsafe actions — and stop at the first failure. Production reality does not stop there: an agent that has already executed an unsafe tool action still faces the consequential questions. Does it notice? Does it contain the damage? Can it roll back state, compensate for the action’s effects, or escalate to a human — and at what cost, in what time, with how much residual risk? This project makes recovery a first-class evaluation object. It injects controlled, sandboxed unsafe-execution events into public benchmark environments, instruments the recovery trajectory that follows, and compares explicit recovery policies (rollback, compensation, escalation) against the implicit baseline of re-prompting — producing the first systematic measurement of how well agents recover, not merely whether they err.

>Sandbox-only research on synthetic failures. Every unsafe-execution event in this project is synthetic, injected by the researcher, inside isolated benchmark environments (RiOSWorld, TheAgentCompany) with reversible operations and synthetic state. No real systems, data, funds, or communications are ever involved. The project describes unsafe-execution scenarios only at the design/evaluation level; it develops no attack techniques and provides no procedures for causing real-world harm.

Who Is This For?

• Agent platform engineers who must specify what their agents should do after an unsafe action — and currently find no evaluation evidence to guide the design
• Safety and reliability researchers for whom recovery is the natural next axis after prevention-focused work
• MPhil/PhD students seeking a thesis with a clean unoccupied question, a bounded sandbox design, and a benchmark contribution
• Benchmark maintainers interested in extending their suites from failure-avoidance scoring to recovery scoring

Research Problem

Benchmarks measure first failures; deployments live with their consequences. An agent that executes an unsafe tool action — sends information to the wrong recipient, modifies state it should not have touched, confirms the wrong transaction — has not finished failing: the run continues, and everything the agent does next either contains the damage or compounds it. The evaluation literature has essentially no measurement of this phase. Benchmarks terminate or score at the unsafe event; no published suite scores whether the agent noticed the error, restored prior state, limited the action’s spread, notified the user, or escalated appropriately. Meanwhile the closest prior mechanism — ToolSafe’s step-level feedback — treats correction as a guardrail feature within an execution loop, not as an evaluated capability with measurable quality dimensions. The research problem is precise: recovery from unsafe tool execution has no definition, no metrics, no benchmark injection methodology, and no comparative evidence on recovery policies, leaving practitioners to design recovery behavior by intuition.

Why This Problem Matters

Prevention will never be perfect, so the quality of the response after failure is a load-bearing property of any deployed agent. The difference between a good and bad recovery is not cosmetic: a fast rollback with correct user notification turns an incident into a log entry; silent continuation after a wrong transfer compounds it. Designers currently choose among rollback, compensation, and escalation with zero evidence about which works when — the exact state of prevention knowledge before the defense-evaluation literature matured. Measuring recovery also reframes safety economics: if explicit recovery policies recover most of the damage from a controlled unsafe action at bounded cost, then system-level safety is the product of prevention and recovery, and evaluations that measure only prevention systematically overstate the risk of systems with good recovery — or understate it for systems without. Either finding changes how agents should be built and scored.

Research Gap

What existing research has demonstrated

The neighboring territory is mapped. RiOSWorld established a risk taxonomy for multimodal agent tasks — a graded classification of which actions are risky and why — providing the principled source of unsafe-action categories for injection design. TheAgentCompany demonstrated long-horizon benchmark execution in reproducible sandboxed environments with rich state, providing the host where recovery behavior can be observed. ToolSafe provided step-level guardrails and feedback loops — the closest mechanism to recovery — establishing that post-error intervention within an execution loop is implementable and measurable. τ-bench contributed the policy-adherence evaluation style that recovery-with-notification naturally extends.

What remains insufficiently studied

What no published work does, per the planning evidence, is treat recovery itself as the evaluation object: no benchmark measures recovery success after unsafe tool execution, no metric suite exists for recovery quality (containment, time-to-recovery, residual risk, notification correctness), no controlled methodology exists for injecting unsafe-execution events that is both safe (sandboxed, reversible, synthetic) and scientifically valid (comparable across agents), and no comparative study contrasts explicit recovery policies — rollback, compensation, escalation — against the implicit default of continuing with re-prompting. The defensible gap statement: risky-action taxonomies and guardrail feedback exist, and long-horizon sandboxed hosts exist, but recovery dynamics after unsafe execution are unstudied as an evaluation dimension.

What Existing Research Shows

• RiOSWorld (Yang et al., NeurIPS 2025) — a risk-annotated multimodal computer-use benchmark whose risky-action taxonomy classifies hazardous agent behaviors by category and severity; supplies the principled unsafe-event categories for injection design. Official page: https://papers.nips.cc/paper_files/paper/2025/hash/0c79d6ed1788653643a1ac67b6ea32a7-Abstract-Conference.html
• TheAgentCompany (Xu et al., NeurIPS 2025) — a reproducible long-horizon benchmark in sandboxed environments with rich synthetic state; the host environment where injected unsafe events and recovery trajectories can be observed end-to-end. Official page: https://neurips.cc/virtual/2025/poster/121705
• ToolSafe (Mou et al., Findings of ACL 2026) — step-level guardrails and feedback for tool-using agents; the strongest published correction mechanism, evaluated here as the closest baseline rather than as recovery itself. Official page: https://aclanthology.org/2026.findings-acl.1850/
• τ-bench (Yao et al., ICLR 2025) — policy-adherence evaluation with simulated users; provides the notification/confirmation protocol style that user-facing recovery steps extend. Official page: https://iclr.cc/virtual/2025/poster/28170

What Remains Open

Four questions connect these findings to this project. First, detection: do agents reliably notice that an executed action was unsafe — or do they continue confidently, treating the error as ordinary task state? Second, policy comparison: which recovery policy family works when — does rollback dominate for reversible state changes, compensation for externally-visible actions, escalation for irreversible ones, and does any single policy dominate overall? Third, dynamics: what are the time and token costs of recovery, and how does residual risk (effects of the unsafe action that recovery fails to undo) distribute across policies and event types? Fourth, interaction: how does recovery behavior interact with task continuation — do agents that recover well also complete the original task, or is there a measurable recovery-versus-completion trade-off? None of these is answered by existing work, because no published evaluation measures the recovery phase at all.

Research Question

> When agents execute unsafe tool actions in sandboxed benchmark environments, which recovery policies — rollback, compensation, or escalation — achieve the highest recovery success at bounded time and token cost, with the least residual risk, and how do explicit policies compare against the implicit re-prompting default?

Research Hypothesis

We hypothesize that explicit recovery policies outperform implicit re-prompting on recovery success and residual risk, that the dominant policy is event-dependent — rollback for reversible state operations, compensation for externally-visible effects, escalation for irreversible ones — and that a policy-selection rule conditioned on event reversibility recovers more value than any single fixed policy. We further hypothesize that detection is the hidden bottleneck: a measurable share of recovery failures will originate not in poor recovery execution but in the agent never recognizing the unsafe event, which re-prompting cannot fix and explicit monitoring can.

Existing Research

• Yang et al., “RiOSWorld,” NeurIPS 2025. Official page: https://papers.nips.cc/paper_files/paper/2025/hash/0c79d6ed1788653643a1ac67b6ea32a7-Abstract-Conference.html
• Xu et al., “TheAgentCompany,” NeurIPS 2025. Official page: https://neurips.cc/virtual/2025/poster/121705
• Mou et al., “ToolSafe,” Findings of ACL 2026. Official page: https://aclanthology.org/2026.findings-acl.1850/
• Yao et al., “τ-bench,” ICLR 2025. Official page: https://iclr.cc/virtual/2025/poster/28170

Proposed Research Direction

The investigation proceeds in four movements. (1) Injection design. Define a controlled unsafe-event catalog drawn from RiOSWorld’s risk taxonomy and instantiated in TheAgentCompany’s sandbox: events span reversibility classes (fully reversible state edits, semi-reversible external messages to simulated recipients, irreversible-but-contained operations), each with a defined ground-truth recovery target — the state and notifications that constitute successful recovery. Injection is researcher-triggered, scripted, and logged; the catalog and triggers are design artifacts of the harness, described publicly at this level only. (2) Instrumentation. Extend the harness to observe the recovery trajectory: post-event state diffs, subsequent tool calls, user-notification events, token and wall-clock costs, and final task state — yielding per-run recovery records aligned to the ground-truth targets. (3) Policy implementation. Implement recovery policies as scaffold components: (a) implicit baseline — the agent continues with its normal loop (re-prompting only what the harness already does); (b) rollback policy — detect-then-restore from checkpointed state; (c) compensation policy — issue corrective actions (simulated recall messages, reversed operations where the catalog defines them); (d) escalation policy — halt and request simulated human confirmation, modeled with coverage and delay parameters from τ-bench-style user simulation; (e) selection policy — choose among the above via an event-reversibility rule. (4) Comparative evaluation. Run the policy × event-class matrix across two agent models with repeated trials, measuring the full metric suite and mapping which policy wins where, at what cost, and with what residual risk.

MPhil Scope

A bounded, executable 6–12 month version: implement the injection catalog for one host environment (TheAgentCompany, RiOSWorld-risk-taxonomy-instantiated), instrument recovery trajectories, and compare three policy families (implicit baseline, rollback, escalation) plus the selection rule across two event reversibility classes, delivering (a) the recovery-benchmark extension with its injection methodology, (b) the first comparative recovery-policy results, and (c) the metric suite and harness code released for reuse. This scope produces one focused contribution — recovery as a measured evaluation dimension — without requiring multi-host saturation or learned recovery policies.

PhD Extension

The doctoral program generalizes along three genuine axes. Recovery-policy learning: replace hand-designed selection rules with learned policies trained on recovery outcomes, and study whether learned policies discover recovery strategies (partial compensation, deferred escalation) that hand-designed rules miss. Containment theory: formalize damage containment — what “residual risk” means operationally, how it bounds under policy classes, and whether recovery guarantees can be stated and checked for restricted action types. Cross-domain transfer: test whether recovery skills learned in one environment transfer to another (TheAgentCompany to RiOSWorld), measure how much injection-catalog diversity recovery evaluation needs, and connect the framework to incident-response practice from systems reliability engineering.

Dataset / Benchmark

• RiOSWorld — risk-annotated multimodal computer-use benchmark. Official page: https://papers.nips.cc/paper_files/paper/2025/hash/0c79d6ed1788653643a1ac67b6ea32a7-Abstract-Conference.html. Relevance: its risky-action taxonomy supplies the principled event-catalog categories and severity gradings the injection design instantiates; used as the taxonomy source and, optionally, a second host. Access: public (proceedings + project release).
• TheAgentCompany — reproducible long-horizon benchmark in sandboxed environments. Official page: https://neurips.cc/virtual/2025/poster/121705 (site: https://the-agent-company.com/, code: https://github.com/TheAgentCompany/TheAgentCompany). Relevance: the primary host — sandboxed, stateful, reversible-friendly, with task structure that makes recovery targets definable. Suitable for MPhil: yes — the extension is scoped to it.
• ToolSafe — guardrail/feedback baseline. Official page: https://aclanthology.org/2026.findings-acl.1850/. Relevance: the closest published correction mechanism, evaluated as baseline.
• τ-bench — simulated-user protocol style. Official page: https://iclr.cc/virtual/2025/poster/28170. Relevance: models the human-confirmation channel for escalation policies.

Baseline Methods

• Implicit continuation (re-prompting default) — the agent’s normal loop with no recovery component; the de facto industry baseline recovery is whatever the agent does by accident.
• ToolSafe-style feedback guardrail — step-level feedback after the unsafe event; the strongest published correction mechanism, included to locate recovery policies relative to the guardrail frontier.
• Checkpoint-restore without detection — rollback applied on a timer regardless of events; isolates the value of detection from the value of restoration.
• Escalation-always — halt and confirm every risky-class action; the maximum-safety/minimum-utility anchor.
• Oracle-detection variants — each policy also run with ground-truth event flags, separating recovery-execution quality from detection quality.

Evaluation Metrics

• Recovery success rate — share of injected events after which the agent reaches the ground-truth recovery target; the headline effectiveness measure.
• Detection rate and detection latency — whether and how quickly the agent notices the unsafe event; the bottleneck diagnostic the hypothesis targets.
• Time-to-recovery and token cost — wall-clock steps and tokens from event to recovered state; the operational cost of each policy.
• Residual risk score — the portion of the unsafe action’s effects unrecovered at run end, graded by the catalog’s severity scale; the safety tail measure.
• Damage containment — the number of subsequent actions affected by the unsafe event before containment; measures compounding.
• Notification correctness — whether, when, and how accurately the user is informed (right event, right severity, right time); the human-facing recovery dimension.
• Task-completion delta — original task success with recovery active versus baseline runs; the utility cost of recovery machinery.

Experimental Design

Environment/dataset: TheAgentCompany as host with the RiOSWorld-taxonomy-derived injection catalog; one open-weights agent model family primary, one second model for robustness; simulated-user channel parameterized from τ-bench style. Baseline: implicit continuation; ToolSafe-style feedback; escalation-always. Proposed method: rollback, compensation, and selection policies with detection components. Test conditions: full policy × event-class (reversible / semi-reversible / irreversible) matrix; with and without oracle detection. Control conditions: no-injection runs per policy (recovery machinery must not damage clean-task performance); sham-injection runs (harness trigger fires, no event) to verify measurement does not itself perturb behavior. Metrics: as listed above. Statistics: repeated trials per cell (n ≥ 5), bootstrap CIs on recovery success and residual risk; paired comparisons across policies on matched event sets; detection-latency distribution analysis. Ablations: detection component removed (oracle-only); notification step removed; checkpoint frequency varied for rollback; selection rule ablated to each fixed policy. Generalization tests: the frozen catalog-instantiation methodology applied to a RiOSWorld task subset as a second host; optionally one additional reversibility class (partially-compensable events) if time allows.

Expected Research Contribution

A potential contribution takes the form of: (a) the recovery-benchmark extension — the first sandboxed unsafe-execution injection methodology with ground-truth recovery targets, (b) the first comparative study of recovery policies for tool-using agents, and (c) a metric suite and released harness that future benchmarks can adopt for recovery scoring. The project does not promise that any policy recovers well — a finding that current agents rarely even detect unsafe events would be a significant negative result that redirects attention to detection, and the design will measure it rather than assume it away.

Limitations

Recovery targets are ground-truth by catalog definition; real incidents involve stakeholder disagreement about what “recovered” means, which the benchmark’s clean definitions do not capture. The event catalog, though taxonomy-derived, is finite — recovery behaviors outside its classes are unmeasured. Escalation is modeled with simulated users (parametric coverage and delay), so findings about human-in-the-loop recovery inherit simulation’s simplifications. The host environments are computer-use and occupational-task sandboxes; recovery in API-only or physical-action settings may behave differently. Detection-quality findings are specific to the studied models and scaffold styles. Finally, injected events are researcher-scripted; naturally-occurring unsafe actions may present different signals than designed ones, and the catalog approach cannot fully anticipate that gap.

Research Ethics and Safety

All unsafe-execution events are synthetic, researcher-injected, and confined to isolated sandbox environments with reversible operations and synthetic state — no real systems, user data, funds, communications, or third-party services are ever involved. The injection methodology is a measurement instrument, not an attack technique: it triggers defined failure conditions the research harness itself defines, and public documentation describes it at design/evaluation level only — no exploit code, no payloads, no procedures for compromising real systems. Escalation channels are simulated; no human subjects receive real notifications. The project’s purpose is defensive: giving the field its first measurements of recovery quality so deployed agents can fail more safely. Findings about detection weaknesses are reported as evaluation results with benchmark context, not as evasion guidance.

Prerequisites

• Python; comfort extending an existing benchmark harness (TheAgentCompany) rather than building from scratch
• Agent architecture literacy: tool-calling loops, state management, checkpointing
• Experimental design and statistics (repeated trials, bootstrap inference, paired comparisons)
• Basic systems skills for sandboxed environment management (containers/VMs)
• Git; disciplined logging for recovery-trajectory data

Suggested Research Timeline

• Phase 1 — Literature & design lock (weeks 1–4): event catalog defined from the risk taxonomy; recovery targets specified.
• Phase 2 — Harness instrumentation (weeks 5–9): injection triggers, state-diff observability, recovery records.
• Phase 3 — Policy implementation (weeks 10–14): rollback, compensation, escalation, selection components.
• Phase 4 — Matrix evaluation (weeks 15–21): policy × event-class × model runs with repeated trials.
• Phase 5 — Detection analysis (weeks 22–24): detection rate/latency deep-dive; oracle comparisons.
• Phase 6 — Generalization (weeks 25–27): second-host application; ablations completed.
• Phase 7 — Analysis & write-up (weeks 28–32): policy-choice evidence, thesis/paper, harness release.

Related Ideas

• Dynamic Pricing Simulator for Sellers — decision reversibility and consequence modeling in a business setting: the same act-then-correct structure in a project-type Idea.
• Kubernetes Cost Optimizer & Rightsizer — rollback-aware infrastructure action: applying changes with the ability to undo them, the operational sibling of recovery policies.
• Investment Portfolio Tracker with Tax-Loss Alerts — alert-driven corrective action in finance, where noticing a problem early and acting deliberately is the whole game.