A research project that answers the question every failed agent run raises: which step broke it? Building an attributed corpus from public long-horizon benchmarks and evaluating prompt-based versus fine-tuned attribution methods that localize the earliest decisive failure step — measured against heuristics and LLM-judge prompting on localization accuracy and attribution lead time.
A research project about the diagnosis gap in agent evaluation. When a long-horizon agent run fails — thirty tool calls, two planning revisions, one environment reset — the evaluation reports a single bit: success or failure. Nobody answers the operational question that follows: which step broke it? Failure attribution exists today as manual trace-reading by experts, an approach that does not scale with agent deployment and leaves benchmarks unable to explain their own results. This project builds what the field lacks: a corpus of agent runs annotated with the earliest decisive failure step, and a systematic comparison of attribution methods — LLM-judge prompting, heuristic rules, and fine-tuned attribution models — evaluated on step-localization accuracy and attribution lead time, the property that makes diagnosis useful before rather than after the run ends.
>Reliability research on public run logs. All experiments use publicly released benchmark logs and harnesses (TRAIL, TheAgentCompany, OSWorld) executed in isolated environments. No attacks are performed and no exploits are produced; the object of study is why agents fail, using failure data that already exists in public benchmarks.
Agent evaluation measures outcomes; debugging requires causes. A long-horizon run fails for one of many reasons — a tool returned an unexpected format three steps ago, a plan was miscalibrated from the start, a state-tracking error compounded, or the environment changed mid-task — and these causes demand different fixes. Yet current benchmarks score the end state and stop there. TRAIL established that expert humans can annotate agent traces with step-level failure information, proving the labels are constructible; it also demonstrated the bottleneck: expert annotation is expensive, and the resulting corpus is small and static. TheAgentCompany and similar long-horizon suites generate thousands of long runs whose failures go unattributed. The research problem is precise: no scalable method exists to localize the earliest decisive failure step in long agent trajectories, and no corpus exists that supports training and evaluating such methods at the scale long-horizon benchmarks produce.
Diagnosis is the bottleneck of the agent-improvement loop. A team that knows “68% of failures originate in tool-result misinterpretation” can prioritize scaffolding changes with evidence; a team that knows only “68% of runs fail” cannot act at all. The same attribution capability upgrades every downstream consumer: benchmarks gain explainable failure statistics, developers gain automated triage, and evaluation gains a diagnostic layer that distinguishes failing-for-hard-reasons from failing-for-fixable-reasons. The timing argument matters too. Earlier attribution — detecting the decisive step while the run is still executing — converts post-mortem analysis into in-flight intervention, the difference between canceling a doomed run and discovering the doom afterward. Whether attribution can work at scale, with what accuracy, and with how much lead time, are unknown quantities the field currently guesses about.
The components exist in isolation. TRAIL provides the first expert-annotated failure-attribution corpus for agent trajectories — proof that step-level failure labels are well-defined and annotatable — while remaining small and static by design. TheAgentCompany demonstrates long-horizon agent evaluation at scale with reproducible runs and rich failure variety, and OSWorld contributes the desktop-agent execution logs that can feed a corpus. AgentAuditor contributes evaluation and analysis structure for agent behavior.
What no published work does, per the planning evidence, is automatic step-level failure attribution: no method has been trained or evaluated to localize the earliest decisive failure step at scale, no attributed corpus exists beyond TRAIL’s expert-annotated set, and no evaluation has compared attribution approaches (prompted judges, heuristics, fine-tuned models) under a common metric suite. The defensible gap statement: expert annotation of failure steps has been demonstrated on a small corpus, and long-horizon benchmarks produce abundant unattributed failures, but the automated bridge between them — scalable, evaluated attribution methods and the corpus to train them — is missing.
Three questions connect these findings to this project. First, feasibility: can current models localize the earliest decisive failure step better than trivial heuristics (blame the last step; blame the first anomaly), when trajectories grow long and failures interact across steps? Second, method: do fine-tuned attribution models outperform prompted LLM judges, and by how much does the margin depend on training-corpus size? Third, lead time: can attribution run during execution with useful accuracy — flagging the decisive step before the run terminates — and how much accuracy does early attribution sacrifice relative to full-trajectory post-hoc attribution? None of these is answered by existing work: TRAIL proves labels exist but not methods at scale; the long-horizon benchmarks provide failures but no attribution.
> Can models be trained or prompted to localize the earliest decisive failure step in long-horizon agent trajectories with accuracy exceeding simple heuristics and unprompted LLM-judge baselines — and how much attribution lead time is achievable without unacceptable accuracy loss?
We hypothesize that trajectory-aware attribution models — trained on step-labeled traces — will outperform both LLM-judge prompting and first-error/last-error heuristics on step-localization accuracy, with the margin growing as trajectory length increases, because judges degrade on long contexts while trained models can learn length-robust failure signatures. We further hypothesize that attribution lead time is substantial: on a meaningful share of failing runs, the decisive failure is identifiable with high confidence several steps before run termination, enabling in-flight intervention that post-hoc-only attribution cannot support.
The investigation proceeds in four movements. (1) Corpus construction. Take TRAIL’s expert-annotated traces as the seed and annotation-schema reference; generate new runs on TheAgentCompany and OSWorld tasks under frozen conditions; annotate a documented subsample for step-level failure labels using the TRAIL schema adapted for long-horizon runs, with dual annotation and agreement measurement on a subsample. (2) Method evaluation. Implement three attribution method families: (a) LLM-judge prompting over full trajectories, (b) heuristic rules (first-error, last-error, anomaly-score-based), and (c) fine-tuned attribution models (sequence models over trajectory encodings). Evaluate all on TRAIL’s test split plus the new corpus. (3) Lead-time analysis. For methods that score incrementally, measure how early in the run each method reaches confident attribution, trading accuracy against detection point. (4) Failure-taxonomy analysis. Classify attributed failures by type — tool-format failures, planning miscalibration, state-management errors, environment-induced failures — and report method performance per class, connecting attribution accuracy to the fix each failure type demands.
A methodological note on the hardest annotation case: long-horizon failures interact. A miscalibrated plan can make a later tool-format error inevitable, raising the question of which step was “decisive.” The annotation protocol handles this explicitly: annotators mark the step at which, had it been different, the run would most plausibly have succeeded — a counterfactual criterion rather than a causal-blame intuition — and cases where two independently sufficient failure points exist are flagged as compound and excluded from the strict localization metric (they remain usable for set-valued attribution analysis). Adjudication of compound flags is documented, and the compound-case rate itself is reported as a corpus property, so downstream users know what fraction of real failures resist single-step attribution.
A bounded, executable 6–12 month version: extend TRAIL with a documented TheAgentCompany run corpus (one model, two task categories, a few hundred runs with subsample annotation), evaluate three method families on localization accuracy, and deliver (a) the extended attributed corpus, (b) the method-comparison results with lead-time analysis, and (c) release of the corpus and evaluation kit. This scope produces one focused contribution — the first systematic, corpus-backed comparison of automated failure-attribution methods — without requiring novel model architectures or multi-benchmark saturation.
The doctoral program generalizes along three genuine axes. Causal attribution theory: move from localization to causal explanation, formalizing when “earliest decisive step” is well-defined under step interactions, and connect attribution outputs to counterfactual re-execution (re-run from the attributed step with a repair) to validate causal claims empirically. Interactive debugging agents: build debugging agents that query the attributed trajectory interactively, requesting targeted re-execution or additional instrumentation — moving attribution from scoring to conversation. Attribution-guided repair: close the loop with automatic repair agents that act on attribution outputs, and measure whether attribution quality measurably improves repair success in a controlled loop.
Environment/dataset: TRAIL test split + extended TheAgentCompany/OSWorld corpus; one primary model for corpus generation. Baseline: heuristics + LLM-judge prompting. Proposed method: fine-tuned attribution over trajectory encodings. Test conditions: trajectory-length stratification (short/medium/long); failure-class stratification; in-flight vs post-hoc attribution conditions. Control conditions: clean-run false-attribution rate (attribution methods must not “find” failures in successful runs); annotation-noise sensitivity via double-annotated subsample. Metrics: as listed above. Statistics: repeated runs (n ≥ 3) for generated runs; bootstrap CIs on localization accuracy; paired comparisons across method families on matched run sets; agreement statistics for annotation quality. Ablations: trajectory-encoding variants (raw logs vs structured representations); corpus-size sensitivity (10%/25%/50%/100% training slices); lead-time/accuracy frontier mapping. Generalization tests: models trained on TheAgentCompany corpus evaluated on TRAIL test and OSWorld traces; optionally cross-model corpus (one additional generator model) to probe attribution robustness to scaffold style.
A potential contribution takes the form of: (a) an extended attributed corpus (the first scaled public corpus for step-level failure attribution), (b) the first systematic method comparison for automated failure attribution, and (c) a lead-time analysis establishing whether in-flight attribution is practically achievable. The project does not promise that attribution accuracy will be high — early results may show current methods far from expert performance, which is itself a valuable negative result that quantifies the diagnosis gap.
Attribution labels encode the “earliest decisive step” judgment, which involves interpretive choices — mitigated by dual annotation and agreement reporting, but the construct is not as objective as end-task success. Corpus extension is scoped to one generator model and two task categories; attribution robustness across scaffolds is tested only partially. The lead-time analysis runs on benchmark tasks, whose step counts, while long, remain shorter than production runs; lead-time conclusions may not transfer to multi-hour deployments. Per-class failure analysis is bounded by the taxonomy’s coverage — rare failure types get thin per-class sample sizes. Finally, attribution methods inherit their training corpus’s scaffold and model idiosyncrasies; cross-scaffold generalization remains genuinely open and is probed only at the PhD scale.
All experiments use publicly released benchmark logs and harnesses executed in isolated environments. No attacks, no exploits, no system compromise: the object of study is why agents fail on their own tasks, using data that already exists in public benchmarks. Failure analysis findings are reported as evaluation research — failure signatures that help debugging are described at the design level, not as a menu of “how to break agents” recipes. No real user systems, credentials, or third-party services are involved at any stage.
Research Idea A research project that makes recovery a first-class evaluation object: injecting controlled, sandboxed unsafe-execution events into agent benchmarks and measuring...
Research Idea A research project that audits the measurement instruments themselves: applying the ABC validity-checklist methodology to agent-security benchmarks to find task-validity...
Research Idea A research project that audits the auditors: measuring how position, verbosity, and self-preference biases distort LLM-judge judgments of safety violations...
Looking for something more accessible? Try these:
Published on September 13, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.
Published on September 13, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.
