TRUSS probes agent skills with controlled execution

Problem

The paper studies automated generation of Agent Skills, which are reusable natural language procedures with executable resources that extend LLM-based agents without changing model parameters. Automatically generating Skills can improve task performance but risks introducing unsafe runtime behaviours that static inspection of the artifact or final task outcomes can miss. The work addresses the dual requirements of Functional Effectiveness (measurable task improvement) and Safety Reliability (absence of security‑relevant or unauthorised actions during execution).

Approach

TRUSS is an iterative, evidence‑guided framework that combines static artifact checks with supervised execution inside a Controllable Execution Environment (CEE). Generation produces a complete Skill package (instructions plus executable resources). A Checker performs static inspection to discharge determinable functional claims and to establish which safety properties require runtime observation. Admitted candidates are executed by a shadow agent in the CEE where brokered tools, an intermediate breaker and a disposable sandbox enforce policies and return provenance‑preserving execution traces. Functional and safety observations are recorded in a Function and Safety Record; failures are returned to a Refiner that edits the Skill and restarts the loop. TRUSS evaluates Skills under task‑conditioned and task‑agnostic regimes and operationalises nine predefined safety properties covering the execution path through which a Skill can influence an agent or its environment.

Key Findings

• Static inspection alone misses a substantial set of runtime vulnerabilities; adding execution evidence improves detection and repair signals.
• Vulnerability detection reached 100.00% precision and recall on 168 SkillInject artifacts (84 matched clean/injected pairs), outperforming LLM-only and static checks.
• Repairing Skills reduced attack success rate on 155 SkillSafetyBench cases from 38.71% to 19.35% with GPT 5.5 and from 46.45% to 29.68% with GPT 5.4, with zero attack regression.
• For Skill generation across 187 SkillGenBench tasks, TRUSS increased task effectiveness from 17.11% (no Skills) to 52.94% and raised benchmark Security from 50.80% to 100.00%, converting unsafe executions into certified safe ones.
• Model behaviour materially affects outcomes: repaired Skills preserved more safe completions under GPT 5.5 than GPT 5.4, showing executor sensitivity.

Limitations

Results depend on the scope of evaluated scenarios and the behaviour of the executor model; TRUSS is sensitive to scenario coverage and executor differences. The framework certifies safety only under the scenarios and observers used during supervised execution; residual unsafe behaviours may remain outside those scenarios. Evaluations used benchmark suites and specific model configurations, so generalisation beyond those setups is not proven.

Implications

Offensive implications arise from the ability to hide multi‑step, state‑dependent attacks inside Skill packages that appear benign under artifact inspection but induce harmful actions at runtime. Attackers can embed download‑and‑execute chains, unauthorised system commands, network interactions or data‑exfiltration flows that only materialise after composed actions. The reported sensitivity to scenario coverage and executor behaviour implies attackers could evade TRUSS by crafting behaviours outside the CEE scenarios or by exploiting quirks of target models, making targeted evasion feasible unless execution coverage and observers are broadened.