A research project that audits the measurement instruments themselves: applying the ABC validity-checklist methodology to agent-security benchmarks to find task-validity flaws, outcome ambiguity, and contamination — then re-running representative agents and defenses on corrected subsets to quantify how much published rankings shift.
A research project about whether the ruler is straight. Agent-security benchmarks — InjecAgent, AgentDojo, WASP, and their peers — are the instruments the field uses to measure attacks and defenses, and their published rankings drive which defenses get adopted. Yet a benchmark is itself a measurement instrument with design choices that can fail: test cases whose “correct” outcome is ambiguous, attacks that a model may have already memorized, task instructions that admit shortcuts that bypass the security property being tested. This project applies the ABC validity-checklist methodology — developed for benchmark auditing in the broader agent-evaluation literature — to the agent-security benchmark family: systematically cataloging validity flaws, re-annotating affected cases with expert agreement controls, and re-running representative agents and defenses on corrected subsets to answer the question that matters: do the published conclusions survive correction, or do the rankings move?
>Meta-science on public artifacts. This project runs no attacks and builds no exploits. It analyzes and re-annotates public benchmark data and re-runs published evaluation harnesses in isolated environments. Its object is measurement validity, not system compromise.
Every claim of the form “defense A outperforms defense B” is mediated by a benchmark, and the validity of that mediation is rarely examined. A benchmark case is valid only if completing it requires the capability it claims to test, its outcome label is unambiguous, and its content has not leaked into model training. Flaws in any layer distort measurement: an ambiguous outcome label lets a judge score a lucky guess as a defense success; a contaminated case measures memorization rather than robustness; a task with a shortcut measures instruction-following rather than injection resistance. The ABC methodology was developed to audit precisely these properties in agent benchmarks. What does not exist is a systematic application of it to the security benchmark family — the benchmarks whose rankings carry real adoption consequences. The research problem is precise: the prevalence, severity, and consequence of validity flaws in agent-security benchmarks are unknown, and no corrected reference rankings exist for any of them.
Instrument error propagates silently. If a nontrivial share of a benchmark’s cases have invalid or ambiguous outcomes, then published attack success rates carry a measurement error band nobody reports, and defense rankings may reflect flaw distribution rather than defensive quality — a defense that happens to fail on invalid cases gets penalized for the benchmark’s bug, not its own weakness. The comparison to other sciences is direct: measurement validity auditing is standard practice where instruments drive decisions. Agent security now has instruments driving decisions — teams pick defenses from these rankings — so auditing them is no longer optional pedantry. There is also a constructive outcome: a documented flaw taxonomy and corrected reference sets make every future benchmark better, and an audit protocol that benchmark authors can run before release prevents the next generation of instrument error. Either way the field learns whether its security evidence is standing on solid ground.
The methodology exists; the application does not. ABC (NeurIPS 2025) established a validity-audit checklist for agent benchmarks — task validity, outcome validity, and contamination as auditable properties with concrete procedures — and demonstrated that audits of this kind are feasible and productive. The agent-security benchmark family itself is mature: InjecAgent provides labeled injection scenarios, AgentDojo provides a dynamic harness with attack/defense hooks, and WASP documents web-agent injection attacks — each published with its own evaluation, none published with an external validity audit.
Existing approaches focus primarily on building and using benchmarks, leaving the audit of the security family unoccupied: no published work applies the ABC checklist systematically across agent-security benchmarks, no flaw taxonomy specific to security evaluation (ambiguity about what counts as a successful attack, judge-dependence of outcome labels, scenario contamination) has been constructed, and no work has measured how correcting validity flaws changes published defense rankings. The defensible gap statement: a validated audit methodology exists, and a benchmark family with high adoption exists, but the two have never been put together — so the validity status of the field’s security measurements is unquantified.
Three questions connect these findings to this project. First, prevalence: what share of cases in each benchmark fail task validity, outcome validity, or contamination checks when audited against the ABC checklist — and which flaw classes dominate in security evaluation specifically? Second, consequence: when representative agents and defenses are re-run on corrected subsets, do published rankings hold, and what is the rank-shift distribution? Third, mechanism: which benchmark-design choices (judge-defined outcomes, scenario templates, public release timing relative to model training cutoffs) predict the flaws, so that future benchmarks can be designed against the failure modes? None of these is answered by existing work, because no published audit of this family exists.
> What is the prevalence and severity of validity flaws — task validity, outcome validity, and contamination — across the major agent-security benchmarks, and how do published attack/defense rankings change when evaluations are re-run on validity-corrected subsets?
We hypothesize that a nontrivial share of cases (on the order of tens of percent in at least one audited benchmark) exhibit at least one validity flaw, with outcome ambiguity around attack-success judgments the dominant class in security suites. We further hypothesize that correcting for validity flaws reorders at least some published defense rankings — specifically that defenses whose reported advantages rest on ambiguous-outcome cases lose more rank stability than defenses evaluated on unambiguous cases — and that contamination effects concentrate in benchmarks whose scenarios were publicly released before common model training cutoffs.
The investigation proceeds in four movements. (1) Audit design. Operationalize the ABC checklist for security benchmarks: define concrete, reproducible checks per flaw class — task-validity probes (does the security property actually gate task success, or can a shortcut satisfy the label?), outcome-validity probes (is the success criterion judge-independent and unambiguous?), and contamination probes (public-scenario overlap with model training eras, memorization-detection heuristics on generated responses). (2) Audit execution. Audit a fixed benchmark set — InjecAgent, AgentDojo, WASP, and one further suite as the generalization target — with two independent annotators per case on audited subsamples, measuring inter-annotator agreement (Krippendorff’s alpha) and resolving disagreements by documented adjudication; produce a per-benchmark flaw registry with severity grades. (3) Corrected re-runs. Re-run a representative agent set (two to three models) and defense set (two to three published defenses) on both original and validity-corrected subsets under frozen conditions; compute the rank-shift of defenses between original and corrected evaluations with uncertainty from repeated runs. (4) Synthesis. Publish the flaw taxonomy, per-benchmark validity profiles, corrected reference rankings, and design recommendations; release the audit protocol so future benchmarks can be pre-audited.
A bounded, executable 6–12 month version: audit three benchmarks (InjecAgent, AgentDojo, WASP) against the operationalized checklist with two-annotator agreement controls, re-run two models and two defenses on original versus corrected subsets, and deliver (a) the flaw registry with prevalence by class, (b) the rank-shift analysis, and (c) the reusable audit protocol. This scope produces one focused contribution — the first validity audit of the agent-security benchmark family — without requiring new benchmark construction or large-scale re-annotation, because the audit targets documented subsamples rather than exhaustively re-labeling every case.
The doctoral program generalizes along three genuine axes. Validity theory for agentic benchmarks: formalize validity constructs for security evaluation specifically (what does “attack success” mean when the agent’s intent is unobservable?), connecting benchmark audit results to measurement-theoretic frameworks and deriving design principles with proofs-of-concept. Automated validity checking: build classifiers and harness-level linters that flag likely validity flaws at benchmark-authoring time, and measure audit-cost reduction against human-audit gold standards. Longitudinal re-audits: re-audit the same benchmarks across versions and model generations, turning validity from a one-time check into a tracked property, and study how contamination surfaces evolve as training corpora and benchmarks co-age.
Environment/dataset: the three public benchmarks, frozen versions with documented hashes. Baseline: published rankings and shipped labels. Proposed method: operationalized ABC audit + corrected re-runs. Test conditions: audited subsamples per benchmark with power analysis for the rank-shift question; full-suite audit where case counts permit. Control conditions: replication runs without correction (isolates variance); agreement calibration on a known-clean seed set. Metrics: as listed above. Statistics: agreement via Krippendorff’s alpha; prevalence via Wilson confidence intervals; rank-shift via paired bootstrap over corrected case sets. Ablations: audit checklist component-wise (task vs outcome vs contamination contributions); annotator-count sensitivity (1 vs 2 vs adjudicated-3). Generalization tests: apply the frozen checklist to one additional security benchmark as a held-out audit; optionally one non-security agent benchmark to probe whether flaw profiles are security-specific.
A potential contribution takes the form of: (a) the first systematic validity audit of the agent-security benchmark family, (b) an empirical meta-science study quantifying how instrument error propagates to defense rankings, and (c) a reusable audit protocol and flaw taxonomy that benchmark authors can apply pre-release. The project does not promise that benchmarks are broken or that rankings will move — it promises the first trustworthy measurement of validity status, with either outcome strengthening how the field evaluates security claims.
Audit coverage is bounded by annotation budget; subsample prevalence estimates carry sampling uncertainty that full-suite audits would not. The ABC checklist operationalization requires judgment calls (what counts as “unambiguous” for security outcomes) that another team might draw differently — mitigated by agreement reporting and protocol release, but not eliminated. Rank-shift conclusions depend on the re-run defense set; findings about two defenses do not generalize to all published defenses. Contamination analysis is inferential (training data is not fully observable), so contamination findings are documented as evidence-backed likelihoods rather than certainties. Benchmarks evolve; audits apply to frozen versions, and version drift may change validity profiles.
This project runs no attacks, builds no exploits, and touches no live systems: its entire substrate is public benchmark data and published harnesses executed in isolated environments. Re-annotation involves reading security scenarios that already exist in public literature. Where audits find flaws, findings are reported constructively with responsible-notification practices to benchmark maintainers ahead of publication. The project’s purpose is measurement integrity — strengthening the instruments on which defensive decisions rely. No operational attack instructions, payloads, or system-compromise procedures are produced at any stage.
Research Idea A research project that makes recovery a first-class evaluation object: injecting controlled, sandboxed unsafe-execution events into agent benchmarks and measuring...
Research Idea A research project that answers the question every failed agent run raises: which step broke it? Building an attributed corpus...
Research Idea A research project that audits the auditors: measuring how position, verbosity, and self-preference biases distort LLM-judge judgments of safety violations...
Ready to level up? These ideas offer more complexity:
Published on September 13, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.
Published on September 13, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.
