MCPZoo finds MCP scanners unreliable at runtime
A new runtime-scale study of Model Context Protocol (MCP) servers shows most security scanners over-flag and disagree. From 64,611 unique servers, 37,288 ran interactively. Scanners marked 96.89% as risky, yet manual checks found less than half of sampled alerts were real. High‑risk tool categories and template reuse amplify attack paths.
Model Context Protocol (MCP) is fast becoming the plumbing that lets Large Language Model (LLM) agents push real buttons: run commands, move files, call services. That makes runtime behaviour the thing that matters. A new study built MCPZoo, the biggest corpus of live MCP servers to date, and then asked the awkward question: do our scanners actually tell the truth?
The team automated the dull and painful bits most of us avoid. They pulled 113,927 entries down to 64,611 unique MCP servers, inferred environments, generated containers, fixed deployment breakage, and validated each server with real protocol interactions across three transports. Of these, 37,288 supported dynamic analysis, with a 57.71% interaction rate and roughly 54% agent success. Reality check: 78.6% of projects lacked a Dockerfile, and only 19.6% of those with complete artefacts ran without modification. This is the ecosystem you are defending.
The attack surface they actually measured
Duplication is rife: 28.3% are duplicates from forks or mirrors. One template shows up 537 times. If I’m red-teaming, I diff that template and go hunting for a single bad default, then spray it across clones. Most servers expose fewer than 10 tools (85.53%), but a few “super-servers” offer thousands. Tool capability is where it bites: 37.66% of tools fall into high-risk buckets and 56.02% are medium. High risk includes command execution, file modification, and outbound data transfer. Chain command exec with outbound transfer and you have an easy exfil path. Land on a super-server and you have reach.
Scanners fail the smell test
Eight popular scanners lit up 96.89% of interactable servers as risky at least once. Sounds scary until you read the fine print: manual validation of a stratified sample puts average precision at 45.53%, with tools ranging from 10.40% to 96.88%. Cross-scanner agreement is a mess. Mean Jaccard similarity is 15.66%, and category-level agreement often falls under 6%. When they checked known-bad cases via mapped CVEs, aggregate detection was 24.17%. One scanner caught 74.29%; several detected none. That gap is an attacker’s friend and a defender’s blindfold.
MCPZoo’s runtime corpus makes this reproducible and queryable. It also spotlights the real stories defenders miss: template-driven replication, fragile deployment, and high-capability tools ready to be chained. There are limits here: the dataset skews to sandboxable projects, scanners and LLM backends matter, and the CVE ground truth is small. But the takeaway is plain enough. If your risk picture depends on a single scanner, you are measuring noise. The interesting question now is whether anyone can get reliable signals from live protocol behaviour at this scale.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability
🔍 ShortSpan Analysis of the Paper
Problem
The paper analyses security risks in the Model Context Protocol (MCP) ecosystem, focusing on MCP servers that expose tools and execute actions for LLM-driven agents. Because many security behaviours manifest only at runtime and real-world servers are hard to deploy, prior assessments relied on small studies or automated scanners whose reliability was unclear. The study asks how the ecosystem behaves in practice and whether existing scanners produce dependable results.
Approach
The authors built MCPZoo, an automated, multi-agent pipeline that collects MCP projects from multiple public markets, infers deployment environments, generates container configurations, iteratively diagnoses failures, and validates servers through real MCP protocol interactions. The framework uses container isolation and a unified interaction gateway to support three transport modes. MCPZoo aggregates 113,927 server entries reduced to 64,611 unique projects, of which 37,288 supported dynamic interaction and were used to evaluate eight representative security scanners under standardised settings.
Key Findings
- Scale and deployability: MCPZoo contains 64,611 unique MCP servers; the framework achieved an overall interaction rate of 57.71% and an agent success rate around 54%. Many projects are not directly runnable: 78.6% lack a Dockerfile and only 19.6% of projects with complete artifacts run without modification.
- Duplication and templates: 28.3% of collected servers are duplicate entities due to forks, copies or mirrors; template-driven replication is common, with the most reused template appearing 537 times.
- Tool exposure and capability: 85.53% of servers expose fewer than 10 tools, but some "super-servers" expose thousands; 37.66% of tools are classified as high risk and 56.02% as medium risk, with high-risk categories including command execution, file modification and outbound data transfer.
- Scanner outputs are unreliable: eight popular scanners flagged 96.89% of interactable servers as risky by at least one tool, yet manual validation of a stratified sample shows average precision of 45.53% (range 10.40% to 96.88%). Pairwise agreement across scanners is low, with mean Jaccard similarity 15.66% and category-level agreement often under 6%.
- Limited recall on confirmed vulnerabilities: using a CVE-based ground-truth, overall detection was low (reported aggregate detection 24.17%), though performance varied by scanner, with one scanner identifying 74.29% of the matched vulnerable servers while several detected none.
Limitations
The dataset favours projects deployable in hermetic sandboxes and so may under-represent servers requiring proprietary credentials or external infrastructure. Scanner selection focused on publicly available tools and specific versions; results depend on scanner configurations and LLM backends. The CVE ground truth is small, covering known issues mapped to MCPZoo entries.
Implications
Attackers can exploit weak deployment practices, replicated templates, and high-capability tools to scale attacks across many visible instances. Low scanner precision and poor cross-scanner agreement mean tooling can both over-report benign capabilities as vulnerabilities and miss confirmed issues, enabling adversaries to remain undetected or to weaponise false confidence. MCPZoo’s runtime corpus and public query interface provide an expanded testbed that attackers could study, but they also supply defenders and researchers with the means to improve scanner reliability and target real exploitable behaviours.
