Post-Hoc Recovery Evaluation: Measuring Agent Recovery After Unsafe Tool Execution
A research project that makes recovery a first-class evaluation object: injecting controlled, sandboxed unsafe-execution events into agent benchmarks and measuring...
A team-oriented product that batch-validates document libraries against PDF/UA and PDF/A rules using the open-source veraPDF engine — prioritized reports, remediation tracking, and honest boundaries about what automated validation can and cannot prove.

A practical product for the teams that ship documents at volume — reports, forms, invoices, policies — and need to know which of their PDFs fail accessibility basics before readers with assistive technology discover it the hard way. The tool batch-validates entire document libraries against the machine-checkable rules of PDF/UA (the accessibility standard for PDFs) and PDF/A using veraPDF, the open-source validation engine built for exactly these standards, then turns raw rule violations into what a team actually needs: a prioritized per-document report, trend tracking across remediation passes, and a clear flag for files that need human review. It is the document-side companion to web accessibility checking — same mission, different format, different rulebook.
>Automated validation identifies standards issues; it does not guarantee accessibility. Passing PDF/UA machine checks does not prove a document is fully accessible to every person or assistive technology — meaningful reading order, sensible alternative text, and comprehensible structure require human judgment. The tool’s job is to find machine-detectable problems at scale and route the rest to people; every report should carry that boundary, and teams should treat results as one input into a real accessibility process, not a certification.
PDF is the default exchange format for institutions — and it is notoriously easy to produce inaccessible PDFs: scans without text layers, documents without tagged structure, images without alternative text, files whose reading order scrambles in a screen reader. The rules for machine-checkable accessibility exist (PDF/UA, with its Matterhorn Protocol of test conditions), and veraPDF implements them as an open-source validator. But a validator answers a single-file question — “does this PDF pass?” — while a document team lives at library scale: which of our 8,000 PDFs are worst, what fails most often, did last quarter’s remediation push work, and which files need a human instead of a rule engine? The gap between a validation engine and a team workflow is exactly the product space this idea occupies.
The product is a batch pipeline wrapped around the veraPDF engine, plus the workflow layer that makes validation results actionable over time.
Teams point the checker at folders, network shares, or a document-management export. The ingester walks the corpus, records each file’s identity (path or content hash), size, and version, and queues it for validation. Re-running against the same corpus skips unchanged files by hash — cheap incremental re-checks are what make library-scale usage pleasant.
Each queued file runs through veraPDF, which parses the PDF and validates it against the selected standard profile — PDF/UA for accessibility, PDF/A for archival conformance, or both — and emits machine-readable validation reports listing every failed rule/check with structured detail. The product runs the engine as a subprocess (or service) and stores the raw report alongside each file’s record. Nothing about the rules is invented: the failure conditions come from the validator’s standards implementation, which is precisely what makes the findings defensible.
Raw reports are per-rule noise; teams need signal. The aggregation layer normalizes violations into a taxonomy: structure (missing tags, broken hierarchy), text (missing document language, empty text layers), media (missing alt text), metadata (missing titles), and scan-suspect (no extractable text at all — a strong signal the file is an image scan needing OCR or human handling). Each document gets a summary: rules failed, categories touched, and a coarse severity rank your team defines (e.g., scan-suspect and structure failures first, since they block assistive reading entirely).
The reporting layer serves three views: per-document (every failed rule with the validator’s detail, plus remediation hints mapped to your authoring tools), library-level (violation categories ranked by frequency, worst-documents list, format/version breakdowns), and trend (scores over validation passes, so a remediation campaign shows up as a curve, not a vibe). Reports export to CSV/JSON for whoever owns the worklist.
Every validation pass is stored with a timestamp and corpus state, so re-validating after fixes produces comparable results. A document’s history shows pass→fail→pass transitions and which categories cleared. This history is the feature that changes behavior: teams stop treating accessibility as a one-time audit and start managing it like technical debt.
Some findings need judgment. The product flags files for human accessibility review: scan-suspect documents (an automated checker cannot assess an image-only PDF’s usability), documents that pass machine checks but carry known-hard content (complex tables, forms), and any file a reviewer marks. The review queue keeps the boundary explicit — automation finds, humans decide — and records reviewer outcomes alongside machine results so the library’s true state is visible in one place.
The long-term win is catching failures at creation. The product exposes a small API (validate a file, fetch a document’s report, webhook on completion) so a CI job can validate PDFs produced by a build, or a CMS plugin can check documents at upload. Same engine, same rules — earlier in the lifecycle, where fixes are cheapest.
A minimal but complete MVP:
Explicitly out of MVP: automated remediation of PDFs, OCR services, CMS plugins, multi-tenant accounts, and authoring-tool integrations. All are natural product roadmap items — none are MVP obligations.
The site’s accessibility and document tooling each target a different object. The AI Website Accessibility Checker audits rendered web pages against web accessibility concerns — DOM, markup, and page-level issues; #064 validates PDF file internals against document standards rules, with batch/history/team workflows a page checker has no need for. The Data Quality Scorecard for Data Teams scores tabular datasets; #064’s subject is documents and its rulebook is a published standard implemented by an open validator. The Container Security Scanner for Docker Images scans image layers for vulnerabilities — a security pipeline; #064 is a standards-conformance pipeline where the “vulnerabilities” are accessibility defects and the fix is a better document, not a patch.
| Tool type | Approach | Limitation |
|———–|———-|————|
| veraPDF (standalone) | The standards engine itself | Single-file CLI/GUI focus; no team workflow, history, or aggregation |
| Commercial PDF suites | Vendor accessibility modules | Per-seat licensing; closed rule reporting; rarely library-trend oriented |
| One-off audit services | Consultant-led document audits | Point-in-time snapshots; no tooling left behind |
| Generic file validators | Format linters in CI | No accessibility semantics; no remediation workflow |
This product’s differentiators: standards-grade open validation wrapped in team-scale workflow — aggregation, prioritization, trends, and human-review routing — with the capability boundary stated on every screen.
Browse more Product Ideas · Intermediate Ideas · Software Development Ideas
A research project that makes recovery a first-class evaluation object: injecting controlled, sandboxed unsafe-execution events into agent benchmarks and measuring...
A research project that answers the question every failed agent run raises: which step broke it? Building an attributed corpus...
A research project that audits the measurement instruments themselves: applying the ABC validity-checklist methodology to agent-security benchmarks to find task-validity...
Ready to level up? These ideas offer more complexity:
Published on September 12, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.
Published on September 12, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.