Covert steering hides inside reusable agent skills

Problem

The paper studies how reusable agent Skills, which are loadable instruction bundles that shape an LLM agent's procedures and decision heuristics, create a supply‑chain risk by acting as externalised behavioural policies. A malicious third‑party Skill can preserve the declared task and valid output interface while covertly redirecting the agent toward an undisclosed objective. The authors formalise Skill Policy Integrity as the requirement that a Skill‑induced policy remain faithful to its declared functionality and the user‑authorised objective; violations occur when a Skill systematically biases decisions without changing task correctness or output format.

Approach

The authors introduce SkillShift, a constrained black‑box framework that searches over a structured, validated space of policy edits appended to a base Skill. A strategy is represented as a tuple of global evaluation principles, task rules, tie‑breaking criteria and semantic examples. Steering mechanisms include policy framing (reweight attributes), tie‑breaking manipulation (plausible secondary criteria) and semantic anchoring (examples to stabilise behaviour). Construction uses an initial plausible strategy from a proposer model, iterative failure‑guided black‑box optimisation using only final outputs, deterministic validators that enforce schema and stealth constraints, category‑level failure patches, and compression to remove redundant or conspicuous content before freezing the Skill. The threat model forbids query, candidate, order or model modification and disallows explicit target commands or instance‑specific shortcuts.

Key Findings

• SkillShift can covertly steer agent choices while preserving valid outputs: in two fixed‑candidate tasks (shopping recommendation and Python dependency selection) it raised attacker‑favoured selection rates to 81.33% (shopping) and 63.33% (Python) while maintaining a 100% valid‑output rate.
• Substantial lift over clean Skills: SkillShift improved selection rates by roughly +44 and +63 percentage points relative to clean Skills in the evaluated domains.
• Frozen strategies generalise: the constructed Skills transferred without further optimisation across held‑out queries, multiple LLM backends and full agent environments. Cross‑model Lifts ranged from about 20.00 to 51.33 percentage points in shopping and 52.67 to 100.00 points in Python; complete‑agent Lifts spanned 28.67–94.00 points while preserving high valid‑output rates.
• Existing scanners largely fail: six evaluated detectors and three registries did not distinguish SkillShift Skills from clean ones under default configurations, whereas an explicit direct override baseline was often detected.
• Downstream utility can degrade: an automated code‑quality judge observed mean code quality falling from 0.558 to 0.506 and reductions in quality‑pass rates, showing valid selection need not imply preserved utility.

Limitations

Evaluation is limited to two fixed‑candidate domains, 50 queries per domain with three runs per query, and specific detector and registry configurations. The optimisation and assessments use particular model endpoints and an automated LLM judge for code quality; results may vary with broader domains, dynamic candidate sets, different evaluators or more exhaustive statistical testing.

Implications

Attackers can covertly bias agent decisions by publishing plausible Skill content that preserves task validity and output format but shifts selection policies. Such Skills are reusable and transferable, enabling supply‑chain or dependency‑level manipulation that is hard to detect with current syntactic scanners. Offensive possibilities include preferential commercial steering, dependency selection bias and subtle long‑term influence of automation workflows. These findings underscore the need for behavioural auditing, provenance and signing of Skills, and counterfactual tests that compare alternative Skill conditions rather than relying on output validity alone.