Benchmark exposes agent skill detector cross-source gaps

Problem

Agent Skills are installable packages that extend LLM agents with instructions, scripts, resources and service configuration. Because a Skill becomes reusable procedural authority inside an agent, Skills form a direct distribution channel for malicious behaviour such as credential theft, manipulation of agent goals, triggered backdoors and concealed unsafe side effects. Existing public resources for malicious Skills are fragmented in format, provenance, evidence strength and benign coverage, and they contain duplicated and structurally related content that complicates aggregation and evaluation. The paper addresses the need for a single, traceable detection benchmark and for evaluation that measures both attack detection and benign over-flagging.

Approach

The authors consolidated 13 frozen public sources, 11 of which met a Core-malicious-artifact rule, and canonicalised heterogeneous records while preserving source and evidence metadata. They deduplicated at exact and conservative text-normalised levels, modelled template-level structural reuse separately, and excluded cross-label conflicts. The frozen primary benchmark comprises 9,740 normalised-unique Skill identities, of which 7,505 are malicious and 2,235 are benign after conservative conflict exclusion. Malicious Core inputs reduced from 8,414 raw records to 7,539 normalised-unique identities organised into 4,588 operational structural families. Using only documented source-native labels and deterministic mappings, they harmonised 11 attack categories for 4,983 malicious identities. For evaluation they trained three learned static text detectors (word TF-IDF with logistic regression and linear SVM, and a character TF-IDF SVM) and compared them with three fixed off-the-shelf Skill scanners under three frozen protocols: Random splits, Malicious-Structural-Disjoint splits and Source-Disjoint (held-out-source) splits. They also ran two robustness controls: class balancing and a narrow scaffold-sanitisation applied before training and evaluation. All experiments used inert static Skill instruction text; no Skill code or payloads were executed.

Key Findings

• Consolidation and audit: 8,414 raw Core malicious records reduce to 7,539 normalised-unique identities and 4,588 structural families; 34 normalised identities conflicted across malicious and benign labels and were removed from the primary benchmark.
• Threat coverage: 4,983 malicious identities were mapped into 11 harmonised attack categories; Execution / Code Delivery is the single largest category in that mapped subset (3,320 identities), with strong source-specific concentration across datasets.
• Learned detector performance: on Random splits Macro-F1 is 0.882 to 0.932; under Source-Disjoint Macro-F1 falls to 0.653 to 0.665. The strongest word TF-IDF SVM scores 0.932 / 0.916 / 0.665 on Random / structural-disjoint / Source-Disjoint. On held-out sources learned models kept high malicious recall (about 94.4 to 95.7%) while benign false-positive rate rose to about 62.0 to 64.4%.
• Off-the-shelf scanners: different operating regimes but unsatisfactory trade-offs. Cisco and SkillSpector had very low benign FPR (about 1.1% and 0.55%) yet detected only around 2.5% and 0% of malicious Skills at frozen gates. SkillFortify achieved higher recall (about 25.3%) at the cost of 49.9% benign FPR. No evaluated detector achieved both high malicious recall and low benign FPR on held-out sources.
• Robustness controls: class balancing and scaffold sanitisation reduced but did not eliminate the Random-to-Source-Disjoint gap, indicating source-conditioned sensitivity beyond simple class imbalance or audited wrapper artefacts.

Limitations

Coverage is bounded by the 13 frozen public sources and by the conservative Core/Main eligibility gates; Main benign coverage is narrower than the malicious side. The benchmark uses static primary Skill instruction text only and excludes package-level or runtime behaviour. Structural families are operational scaffold groupings and do not imply lineage, actor identity or campaign membership. Source-Disjoint is a measure of source-conditioned generalisation with provenance, construction and label composition coupled and therefore is not a causal estimate of universal unseen-source difficulty. Five malicious records with sensitive credentials were released in sanitized form.

Implications

From an offensive perspective, Agent Skills provide a direct channel to distribute malicious capabilities that can be trusted by agents once installed. The findings suggest attackers can exploit cross-source heterogeneity and scaffold reuse to proliferate malicious Skills and to evade or subvert static detectors: they can craft Skills that mimic held-out source conventions to induce high false positives, exploit scanner operating points to avoid sensitive scanners, or reuse templates to amplify distribution. Because learned detectors tend to over-flag unfamiliar benign conventions while fixed scanners often trade recall for low false positives, an adversary can either attempt to evade detection or weaponise over-flagging to cause denial or disruption of legitimate Skills. These dynamics underscore that practical attacks can combine content, provenance and packaging choices to affect both detection and benign over-flagging.