Ecosystem-Scale Measurement of Tool-Poisoning Risks in MCP Server Registries

A research project about counting what the agent ecosystem has only ever illustrated. The Model Context Protocol (MCP) has become the standard way LLM agents discover and invoke external tools, and security researchers have demonstrated that tool descriptions can carry hidden instructions — text the user never sees but the model consumes — along with cross-origin shadows and descriptor bait-and-switch tricks. Those demonstrations, however, are anecdotal: a handful of crafted examples, not an ecosystem estimate. This project builds the missing measurement: a corpus of publicly listed MCP servers and their tool metadata, a labeling methodology for poisoning-pattern classes, quantified prevalence per class, and the registry-level controls the numbers imply. The deliverable is a measurement study with a released corpus and the first evidence base for MCP marketplace governance.

>Measurement-only research on public metadata. The project crawls and analyzes publicly listed tool descriptions — the same information any user’s agent downloads. It does not invoke third-party tools, does not execute third-party code, does not contact vendors beyond normal listing access, and handles any discovered poisoning through coordinated disclosure, not exploitation.

Who Is This For?

• MPhil students seeking a measurement thesis with low compute and a concrete artifact (corpus + analysis)
• Security researchers in supply-chain and marketplace security moving to the agent-tool ecosystem
• MCP registry/marketplace maintainers who need prevalence data to justify vetting controls
• Policy and governance researchers tracking how agent-tool ecosystems get secured

Research Problem

An agent that connects to an MCP server trusts that server’s descriptions — the text telling the model what each tool does and when to use it. Security work has shown descriptions can hide instructions visible to the model but not the user (tool poisoning), can shadow higher-trust tools across origins, and can be swapped after initial review. But the published record consists of demonstrations and threat taxonomies; nobody knows whether these patterns are rare curiosities or common practice across the registries agents actually browse. The research problem: there is no ecosystem-scale, labeled measurement of tool-poisoning patterns in public MCP listings — so registry operators, agent vendors, and users are making trust decisions with no prevalence data at all.

Why This Problem Matters

Marketplaces without measurement repeat history. Public package registries spent a decade discovering that typosquatting, maintainer takeover, and install-script abuse were measurable, predictable, and controllable only after systematic measurement made them visible. MCP registries are earlier in that curve and higher-stakes: a poisoned tool description is consumed by a model with instructions and data access, not a human reading release notes. Prevalence numbers per pattern class determine everything downstream — whether registries need automated scanning at intake, whether agent runtimes need description sanitization, whether cross-origin shadowing needs explicit protocol controls. Without the measurement, every governance decision is a guess; with it, controls can be prioritized by the risk they actually address.

Pattern Classes Under Measurement

The codebook operationalizes the peer-reviewed threat taxonomy into four measurable classes. Hidden instructions — description text containing imperative content directed at the model rather than the user (instructions to exfiltrate, prefer, or ignore), invisible in normal UI rendering but consumed by the agent. Cross-origin shadows — tool names or descriptions positioned to shadow higher-trust tools from other servers (naming similarity to official tools, claimed override relationships), the pattern class the threat model identifies with malicious-server connection vectors. Rug-pull descriptors — indicators of post-review description volatility, measured longitudinally where snapshots permit: descriptions that change materially after adoption windows. Capability mismatches — claimed capabilities diverging from structural evidence (declared scopes, requested permissions, category norms), the softest class and the one where inter-annotator agreement is most informative. Each class carries explicit inclusion/exclusion criteria in the codebook, a decision tree for borderline cases, and worked examples; the agreement statistics reported in the study are per class, because a codebook that is reliable for hidden instructions may be unreliable for capability mismatches, and consumers of the measurement need to know which numbers carry which uncertainty.

Research Gap

What existing research has demonstrated

Academic threat modeling of MCP has catalogued the attack surface: the peer-reviewed threat-modeling paper (MDPI Cybersecurity 2026) systematizes MCP threats including malicious servers providing poisoned tool descriptions (MCP-C01 in its classification), and industry security work (Invariant Labs, April 2025 — supplementary, non-peer-reviewed, used here only as context) demonstrated hidden-instruction tool poisoning concretely. Defense-side work on tool-call safety (ToolSafe, Findings of ACL 2026) addresses runtime guarding of tool invocation. Ecosystem-scale vulnerability-scanning methodology exists from package-registry security research.

What remains insufficiently studied

Existing approaches focus primarily on threat characterization and runtime defense, leaving ecosystem prevalence measurement insufficiently studied: no published work quantifies how frequently poisoning patterns occur across public MCP listings, no labeled corpus of tool descriptions exists for training or auditing detectors, and no work evaluates which registry-level controls would address the observed distribution of risks. The measurement gap is also temporal — the ecosystem is growing quickly, and a baseline study enables the longitudinal work that package-registry security shows is essential. Defensible statement: prior studies demonstrate that tool poisoning is possible and model it as a threat class, but its actual prevalence across public registries remains unmeasured.

What Existing Research Shows

• MCP threat modeling (Huang et al., MDPI Cybersecurity 2026) — systematized MCP threat taxonomy including malicious-server connection enabling tool-poisoning attacks via poisoned tool descriptions; establishes the pattern classes worth measuring.
• Tool-poisoning demonstrations (Invariant Labs, April 2025 — industry source, supplementary evidence) — concrete hidden-instruction examples in tool descriptions; motivates the measurement without substituting for it.
• ToolSafe (Mou et al., Findings of ACL 2026) — runtime step-level guardrails; shows the defense side of tool-call safety and why description-level risk is one of the surfaces such guards watch.
• WASP (Evtimov et al., NeurIPS 2025) — end-to-end injection-surface framing for agents; methodological precedent for measuring injection exposure rather than only demonstrating it.

What Remains Open

Three questions turn the demonstrations into science. First, prevalence: what fraction of public MCP listings exhibits each pattern class (hidden instructions, cross-origin shadows, rug-pull descriptors, misleading capability claims), and how does that concentrate by category, maintainer, or privilege level? Second, detectability: how well do automated auditors — pattern scanners versus LLM-based description analyzers — identify these patterns against expert labels, which determines whether intake-time scanning is viable? Third, control mapping: which registry controls (intake scanning, description-provenance signing, change-notification windows, cross-origin shadow detection) address the measured distribution at acceptable cost? The published literature answers none of the three.

Research Question

> What is the prevalence of tool-poisoning pattern classes across public MCP server listings, how reliably can automated auditors detect them against expert labels, and which registry-level controls map onto the measured risk distribution?

Research Hypothesis

We hypothesize that a measurable minority of public listings exhibits at least one poisoning pattern, with concentration in high-privilege categories (filesystem, shell, payments) and among tools that shadow established services; that LLM-based auditors achieve useful but imperfect precision against expert labels, with pattern-scanner precision varying strongly by class; and that the measured distribution implies intake-time scanning plus provenance signaling as the highest-yield controls.

Existing Research

• Huang et al., “Model Context Protocol (MCP): Landscape, Security Threats, and Mitigation Approaches” (threat modeling), MDPI Cybersecurity, 2026. Official page: https://www.mdpi.com/2624-800X/6/3/84
• Invariant Labs, “MCP Security Notification: Tool Poisoning Attacks,” April 2025 (industry demonstration — supplementary evidence, not peer-reviewed). Reference: https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks
• Mou et al., “ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback,” Findings of ACL 2026. Official page: https://aclanthology.org/2026.findings-acl.1850/
• Evtimov et al., “WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks,” NeurIPS 2025. Official page: https://neurips.cc/virtual/2025/poster/121728

Proposed Research Direction

The investigation is a measurement study in four stages. (1) Corpus construction. Crawl public MCP registry listings (server metadata, tool names, descriptions, capability declarations) respecting listing terms; snapshot ethically (rate-limited, dated, versioned) and document corpus construction so it is repeatable. (2) Labeling methodology. Operationalize the threat taxonomy’s pattern classes into codeable rules (hidden-instruction heuristics, shadow-description relations, capability/description mismatch); expert-label a stratified sample (by category and popularity) with double annotation and disagreement adjudication; report inter-annotator agreement. (3) Prevalence estimation. Estimate per-class prevalence over the labeled sample with confidence intervals; extrapolate carefully to the full corpus with documented assumptions; analyze concentration by category, privilege, and maintainer signals. (4) Auditor evaluation and control mapping. Evaluate pattern-based scanners and LLM-based auditors against expert labels (precision/recall per class); map the measured distribution to candidate registry controls with a qualitative cost/benefit analysis, and coordinate responsible disclosure for any confirmed malicious findings. Model choices: one open-weights LLM for the auditor family, documented; scanning infrastructure released with the corpus.

MPhil Scope

A bounded 6–12 month version: crawl one major public registry into a versioned snapshot; expert-label a stratified sample of several hundred listings; estimate prevalence for three to four pattern classes with confidence intervals; evaluate one scanner and one LLM auditor against the labels; deliver the corpus (scrubbed of weaponizable content), the codebook, and the analysis. One focused contribution: the first labeled prevalence baseline for MCP tool poisoning — feasible on a student budget because the object of study is metadata, not model runs.

PhD Extension

The doctoral program turns the baseline into longitudinal ecosystem science. Longitudinal measurement: repeated snapshots tracking poisoning-pattern dynamics, rug-pull events (description changes after adoption), and registry-policy effects over time. Cross-registry comparison: extend across registries and marketplaces to test whether governance structure predicts risk distribution. Detection science: train dedicated poisoning-detection models on the accumulated labeled corpus, evaluate adversarial description-writing against them, and develop certified description-integrity mechanisms (signing, provenance, canonical hosting) evaluated at ecosystem scale. Deployment study: partner with a registry to A/B intake controls and measure their effect on the measured prevalence.

Dataset / Benchmark

• Public MCP registry listings — the measurement corpus itself (server metadata, tool descriptions, capabilities). Official source: the public registries’ own listing pages/APIs; the snapshot is constructed by the project and released in scrubbed form. Relevance: no equivalent labeled corpus exists; constructing it is the contribution. Suitable for MPhil: yes — crawling and labeling are compute-light; the work is methodological diligence.
• MCP threat taxonomy (MDPI Cybersecurity 2026) — the pattern-class definitions the codebook operationalizes (https://www.mdpi.com/2624-800X/6/3/84). Relevance: ensures measured classes correspond to the peer-reviewed threat model rather than ad-hoc categories. Suitable for MPhil: yes, as the coding frame.
• No benchmark scores or dataset sizes are asserted for registry contents — all quantities are what this study itself measures and reports with uncertainty.

Baseline Methods

• Pattern-based scanner — deterministic heuristics from the codebook (instruction-like phrasing, shadow-name similarity, capability mismatch), the auditable low-tech baseline.
• LLM-based description auditor — a frozen open-weights model prompted with the codebook to flag suspect descriptions, representing the scalable automated approach.
• Expert labels — the ground-truth reference for both auditors, produced with double annotation on the stratified sample.
• Null model — prevalence expected if patterns were randomly distributed across categories, for concentration testing.

Evaluation Metrics

• Per-class prevalence with binomial confidence intervals on the labeled sample.
• Concentration statistics — prevalence by category/privilege tier versus the null model.
• Auditor precision, recall, and F1 per pattern class against expert labels.
• Inter-annotator agreement (e.g., kappa) for the codebook’s reliability.
• Corpus coverage statistics — snapshot completeness and versioning quality, reported as measurement validity indicators.

Experimental Design

Environment/dataset: one public registry snapshot (primary); a second registry snapshot if access terms permit (generalization). Baseline: pattern scanner vs LLM auditor vs expert labels. Proposed method: the codebook + labeling methodology + prevalence estimation pipeline itself. Test conditions: stratified sampling across categories and popularity tiers; double annotation with adjudication. Control conditions: null-model concentration tests; re-labeling of a subsample after a time gap to estimate codebook stability. Metrics: as above. Ablations: codebook-rule variants (strict vs lenient hidden-instruction criteria) to test classification sensitivity. Generalization tests: cross-registry comparison where terms permit; cross-time re-snapshot for early longitudinal signal.

Expected Research Contribution

A potential contribution includes: the first labeled prevalence measurement of tool-poisoning patterns in public MCP listings; a released corpus and codebook enabling detection research; an auditor evaluation establishing whether automated intake scanning is viable; and an evidence-based control-mapping analysis for registry governance. Because the study measures rather than asserts, its conclusions follow from the data — including the possibility that prevalence is low, which would itself be a load-bearing positive finding for the ecosystem.

Limitations

Public listings under-report what servers actually do at runtime; description-level measurement cannot see server-side behavior changes (rug-pulls) except through longitudinal re-snapshots. Sampling coverage depends on registry access terms, which may constrain corpus breadth or prevent cross-registry comparison. Labeling is judgment-bound despite the codebook; agreement statistics quantify but do not eliminate subjectivity, especially for borderline “misleading capability claims.” The study measures the listed ecosystem — private/enterprise MCP deployments are out of scope. Auditor evaluation inherits LLM limitations; auditor errors on ambiguous descriptions are expected and reported rather than hidden. Responsible disclosure may remove some measured content mid-study, which is handled as documented attrition.

Research Ethics and Safety

The project analyzes publicly listed metadata under registry terms of access — the same data any agent’s discovery phase downloads. It does not execute third-party code, invoke third-party tools, or attempt exploitation of any kind. Discovered poisoning is handled by coordinated disclosure to the registry/maintainer with publication deferred or scrubbed per responsible practice. The released corpus is scrubbed so it cannot function as an attack recipe collection. The research objective is protective: giving registry operators and agent vendors the first prevalence evidence base for governance decisions.

Prerequisites

• Python (crawling, data analysis); familiarity with APIs and rate-limit etiquette
• Statistics for prevalence estimation and concentration testing
• Basic NLP/LLM tooling for the auditor family
• Security fundamentals (supply-chain threat models)
• Ethics discipline for measurement research (terms of service, disclosure practice)

Suggested Research Timeline

• Phase 1 — Literature & codebook (weeks 1–4): taxonomy operationalization, pilot labeling, codebook v1.
• Phase 2 — Corpus construction (weeks 5–8): crawl, snapshot versioning, stratification design.
• Phase 3 — Labeling (weeks 9–14): double annotation, adjudication, agreement analysis, codebook v2.
• Phase 4 — Prevalence estimation (weeks 15–18): per-class estimates, concentration analysis.
• Phase 5 — Auditor evaluation (weeks 19–22): scanner + LLM auditor vs labels.
• Phase 6 — Control mapping & disclosure (weeks 23–25): governance analysis; coordinated disclosure where needed.
• Phase 7 — Write-up & release (weeks 26–28): thesis/paper; scrubbed corpus + codebook release.

Related Ideas

• Container Security Scanner for Docker Images — the site’s supply-chain security sibling: scanning an ecosystem artifact for risk before it runs.
• Data Quality Scorecard for Data Teams — measurement-and-scoring methodology applied to a different artifact class.
• AI Website Accessibility Checker — an AI-audit-of-public-content precedent with automated-checker-plus-human-review structure.