Reproducible Benchmark Harness for Desktop AI Agents

A research project about measuring AI agents rather than building them. Computer-use agents — models that click, type, and navigate real applications — are usually demonstrated on cherry-picked successful runs. This project builds the opposite: a small, reproducible evaluation harness in which desktop agents execute defined tasks inside isolated sandboxed environments, every step is logged, and success is decided by executable checkers (the actual file state, the actual application state) rather than vibes. The design follows the methodology of open execution-based benchmarks such as OSWorld — task definitions, environment setup scripts, and programmatic success checks — and the project’s deliverable is a working harness plus a rigorous failure analysis of one or more agents on your own task set.

>An evaluation harness, not an autonomous agent product. Everything here runs agents inside disposable, isolated environments against synthetic data — never on your daily system, never with real credentials, and never against production services. The harness exists to produce trustworthy measurements; the isolation discipline that makes measurements trustworthy is the same discipline that keeps the exercise safe.

Who Is This For?

• ML engineers who need to evaluate or compare computer-use agents honestly
• QA and test-automation researchers bringing software-testing rigor to agentic systems
• Graduate students seeking a research project with a clear artifact: a reproducible harness and a results paper
• Builder-evaluators who want to understand why agents fail, not just whether they pass

The Problem

Agentic AI demos are easy to stage and hard to trust. A screenshot of a successful run proves almost nothing: the task may have been hand-tuned, the environment pre-arranged, the failures cropped out. Real evaluation requires the same machinery that made software benchmarks trustworthy — precisely defined tasks, controlled environments, automated setup, execution-based scoring, and complete traces that let a third party replay the experiment. The OSWorld benchmark (from XLANG Lab, published at NeurIPS 2024) demonstrated this methodology at scale: hundreds of real cross-application tasks on Ubuntu/Windows/macOS environments, scored by checking the resulting state, not the agent’s self-report. What a learner or small team usually lacks is a harness of their own — a small, hackable version of that discipline for the handful of tasks and agents they actually care about, with every design decision visible.

How It Works

The project separates concerns the way serious benchmarks do: task definition, environment provisioning, agent execution, instrumentation, and scoring are five independent pieces joined by explicit artifacts. Build them in that order.

1. Define Tasks as Data, Not Stories

A benchmark task is a structured record, not a prompt. Each task specifies: the goal statement given to the agent, the starting environment state (files present, app state, configuration), the allowed interaction surface, a time/step budget, and — critically — one or more executable success checkers that inspect final state (a file exists with expected content; a document was exported; a setting was changed). Write ten to twenty tasks across a few application families (file manager, text editor, spreadsheet, terminal), each with a stated intent (e.g., “locate and rename a file according to a rule in a note”). Store tasks as versioned data files so every experiment names exactly which task set it ran.

2. Provision Sandboxed Environments

Each run starts from a clean, disposable environment — a container or virtual machine image restored from a known snapshot. Setup scripts materialize the task’s starting state deterministically: seed files with fixed content, fixed application configuration, no network access unless a task explicitly requires a controlled local service. Ephemeral, single-use environments are the harness’s backbone: no state leaks between runs, and any run can be reproduced by replaying the same setup against the same snapshot.

3. Execute the Agent Under Instrumentation

Run the agent against the environment with a step budget and record everything: the task version, agent identity and configuration, every observation it received, every action it took, timestamps, and resource usage. Instrumentation is capture-first — you are building the dataset you will later analyze. Traces are stored as structured records (one per step) alongside the environment’s own logs, so post-hoc analysis can distinguish “the agent clicked the wrong thing” from “the application never rendered the thing.”

4. Score with Execution-Based Checkers

When a run ends (task completed, budget exhausted, or agent gave up), the harness runs the task’s checkers against the actual environment state. Checkers return structured verdicts — pass, fail, with details — never a model’s claim about itself. Where partial credit is meaningful (a file found but not renamed), define graded outcomes in the task definition so scoring stays mechanical and re-runnable.

5. Categorize Failures

The research heart of the project. Convert raw failed traces into categories: perception failure (agent misread the screen), planning failure (right reading, wrong sequence), execution failure (right plan, wrong click/keystroke), environment mismatch (setup differed from task intent), and timeout. Build a small labeling tool — even a spreadsheet workflow over exported traces — and produce a failure-distribution report per task and per agent. This report is the artifact that distinguishes a harness project from a leaderboard vanity run.

6. Aggregate and Report

Persist every run in a results database (run, task, agent, config, verdict, failure category, duration, steps). Reporting joins across runs: success rate per task and per agent, pass-rate variance across repeated runs of identical configuration (reproducibility itself is a measured property!), and per-category failure breakdowns. Export a summary report that a stranger could audit: which tasks, which agents, which configurations, which numbers.

7. Reproduce or It Did Not Happen

The harness’s final test of itself: replay a sample of runs from stored configs and snapshots and verify the recorded verdicts reproduce. Document any that do not — nondeterminism in agents is a finding, not an embarrassment, as long as it is measured and reported.

Key Features

• Tasks-as-data with versioned definitions, deterministic setup scripts, and executable checkers
• Disposable sandboxed environments — snapshot-restored containers/VMs, single-use, no residual state
• Full step-level instrumentation — observations, actions, timestamps, and environment logs per run
• Execution-based scoring — verdicts from environment state via mechanical checkers
• Failure taxonomy tooling — structured categorization of perception/planning/execution failures
• Run registry and reproducibility checks — every number traceable to a stored configuration and replayable

Functional Requirements

• Load task definitions from versioned data files with schema validation
• Restore a clean environment snapshot and apply a task’s setup script before each run
• Execute an agent within a step/time budget while capturing a complete structured trace
• Run execution-based checkers post-run and store structured verdicts
• Store all runs in a database with task/agent/config provenance
• Produce aggregate reports: success rates, failure categories, cross-run variance
• Support replay of stored run configurations for reproduction verification

User Stories

• As an ML engineer, I want pass rates per task per agent computed from executable checks so I can compare systems without trusting their self-reports.
• As a researcher, I want failure-category distributions so my paper can say why agents fail, with trace evidence.
• As a QA researcher, I want to replay a run from its stored configuration so I can verify reported numbers independently.
• As a student, I want to add a new task by writing one data file plus one checker — no harness code changes.
• As a skeptical reviewer, I want repeated identical runs with recorded variance so stability is a measured property, not a hope.

MVP Scope

A minimal but complete MVP:

Explicitly out of MVP: multi-agent comparison at scale, graphical-task generation, cloud-device farms, web-browser agents, and any production-agent development. Extensions, not obligations.

Project Timeline

• Weeks 1–2: environment snapshots and setup scripting; first three tasks end-to-end (define → run → check)
• Week 3: full task set; trace capture hardened; results database
• Weeks 4–5: agent runs across the task set; failure categorization; variance study across repeated runs
• Week 6: aggregate report, reproducibility replay, write-up with threats-to-validity section

Testing Strategy

• Setup determinism tests: applying a task’s setup twice to a fresh snapshot yields byte-identical state
• Checker tests: checkers pass/fail correctly against hand-constructed success and failure states; no checker depends on agent behavior
• Isolation tests: a run cannot write outside its sandbox; network-isolation assertions where configured
• Replay tests: stored run configs re-execute and produce the recorded environment verdict (agent nondeterminism documented separately)
• Instrumentation tests: every run’s trace is complete (no missing steps), schema-valid, and joins cleanly to the results database

Research Ethics and Data Considerations

• Isolation is non-negotiable: agents execute only inside disposable sandboxes — containers or VMs restored from clean snapshots — never on a host system or any machine holding real data
• No real credentials, ever: environments use synthetic accounts and synthetic documents; a task that seems to need “a login” gets a fake one
• Network discipline: default-deny networking inside sandboxes; where a task needs a service, run a local, controlled stub — never production endpoints
• Synthetic data only: seed documents, files, and records are fabricated for the benchmark; nothing personal or confidential enters an environment
• Controlled permissions: the agent’s in-environment account gets the minimum rights its tasks require; no security-control bypass is ever in scope
• Report honestly: publish your task versions, agent configs, and failure criteria with results; an unshareable benchmark result is a marketing claim, not research

Limitations

• Task coverage: ten to twenty hand-written tasks measure those tasks, not general computer-use ability; scope claims must match the task set
• Agent nondeterminism: identical configurations can produce different runs; treat repeated-run variance as part of the result, not noise to hide
• Environment fidelity: containerized app families approximate, but do not exhaust, real desktop diversity; findings may not transfer across environments
• Checker blind spots: executable checks verify state that is cheap to verify; qualities like “reasonable formatting” may need human review, which should be labeled as such
• One-harness bias: your harness’s own quirks (snapshot timing, budget limits) shape results; the threats-to-validity section exists to say this out loud

Success Metrics

• The full task set runs end-to-end unattended, with zero environment state persisting between runs
• Every reported number traces to stored runs with task/agent/config provenance and can be replayed
• Failure categories cover ≥90% of failed runs with trace-level evidence
• Repeated identical runs quantify pass-rate variance, and the report interprets it
• A stranger can add a new task by writing one definition file plus one checker, guided by the README

Common Challenges

• Snapshot drift — environment images mutate as you develop; version snapshots with the task set that references them
• Checker temptation — writing checks that mirror what the agent usually does instead of what the task requires; derive checkers from the task’s stated intent
• Flaky automation — UI timing inside sandboxes makes steps flaky; budget retries explicitly and report them rather than absorbing them silently
• Failure ambiguity — some runs fail for compound reasons; allow primary/secondary categories instead of forcing one label
• Scope creep into agent-building — the deliverable is the harness and the analysis; improving the agent under test is a different project

Learning Objectives

• Internalize execution-based evaluation: why state-verified scoring is the standard for agentic benchmarks
• Design reproducible experiments — versioned tasks, deterministic setup, replayable runs — as engineering artifacts
• Build instrumentation-first systems where every run generates analyzable data
• Practice failure analysis as a research skill: taxonomies, evidence, distribution reports
• Handle potentially powerful software (agents) with disciplined isolation and explicit ethical boundaries

Why This Idea Is Different

This project measures agents; its neighbors on this site each do something else with adjacent vocabulary. The AI Test Case Generator from Source Code generates tests for human-written code — a code-comprehension deliverable; #063 generates no tests at all, it administers tasks to autonomous agents and scores outcomes. The Container Security Scanner for Docker Images inspects image contents for vulnerabilities — a security-analysis pipeline; #063 uses containers purely as disposable evaluation sandboxes, not as scan targets. The Data Quality Scorecard for Data Teams scores datasets against quality rules; #063’s checkers score agent behavior against task intents. Same rigor family, entirely different objects of measurement.

What Similar Tools Exist

| Tool type | Approach | Limitation |
|———–|———-|————|
| OSWorld-scale benchmarks | Hundreds of cross-platform tasks, community-run | Heavyweight for a small team; fixed task sets not tailored to your agents |
| Vendor agent eval suites | Proprietary evaluations bundled with agent products | Closed tasks and scoring; self-grading conflict of interest |
| Ad-hoc demo scripts | Single-task showcases | No isolation, no checkers, no reproducibility |
| General CI frameworks | Run anything on a schedule | No environment snapshotting, task semantics, or failure taxonomy for agents |

This project’s differentiators: a learner-scale harness with the same methodological skeleton as public benchmarks, execution-based scoring, built-in reproducibility replay, and a failure-taxonomy analysis that turns runs into research findings.

Technology Stack

• Python — harness orchestration and tooling
• Docker (with snapshot/restore workflow) or a VM runner — disposable environments
• pydantic / JSON Schema — task-definition validation
• SQLite — results database
• pandas + matplotlib — aggregation and failure-distribution reports
• pytest — checker, setup-determinism, and isolation tests

Future Enhancements

• A second and third agent under test for genuine cross-agent comparison
• Web-family tasks behind a controlled local mock site
• Human-review labeling for qualities executable checkers cannot capture, with inter-rater agreement reported
• Export of the failure-taxonomy tooling as a reusable small library
• A published mini-report replicating one public benchmark finding at learner scale, with citations

Related Ideas

• AI Test Case Generator from Source Code — generating tests for human code, where this project scores autonomous agents
• Container Security Scanner for Docker Images — containers as security-scan targets, where here they are disposable evaluation sandboxes
• Data Quality Scorecard for Data Teams — dataset quality scoring, sharing this project’s measurable-criteria discipline

Browse more Research Ideas · Advanced Ideas · AI Ideas