CS-Guard finds LLM guardrails leak malicious code

Problem

This paper evaluates how well existing guardrails prevent large language models from generating malicious code. As LLMs are increasingly used as coding assistants, adversaries can exploit them to produce malware. The paper addresses a gap: there is no standard, fine-grained benchmark that measures guardrail effectiveness specifically in code-generation settings, including both natural-language-to-code and code-to-code interactions, nor for stealthy attacks that hide malicious intent in plausible development scenarios.

Approach

The authors introduce CS-Guard, a benchmark that tests nine representative guardrails across seven LLMs in two settings. Text-to-code: 1000 high-quality malware-generation prompts derived from expert TTP descriptions, augmented with seven jailbreak styles and a novel fictional scenario attack (FSA) that embeds malicious objectives within legitimate software-development scenarios. Code-to-code: 331 prompts covering code infilling, code completion and code translation collected from malware-source benchmarks. CS-Guard organises guardrails in a modular three-layer taxonomy (category, operation, position) to permit modular evaluation of strategy-type, classifier-type and internal-type defences. Evaluations label model outputs as good, bad or unclear using an LLM labeler validated against human annotators. Metrics include attack success rate (ASR) for input and internal guards and F1 for output classifiers. Single-turn and a pilot multi-turn FSA variant are included; experiments sample multiple stochastic generations and count a prompt as successful if any generated response is malicious.

Key Findings

• Overall poor guardrail effectiveness: many methods fail to stop malicious code generation. For text-to-code, average ASR after jailbreaks is about 50% for many guardrails.
• Internal-type model vulnerabilities: classic models exhibit very high ASRs (for example, two models reached about 98% and 94.5% on level-1 text-to-code), while state-of-the-art open models still showed nontrivial ASRs (15.1%–61.4%). FSA yields ASRs of 85.9%–99.5% against internal-type models.
• Code-to-code is especially fragile: average ASRs on base models ranged from roughly 52.5% up to 98.9%; models were relatively more resistant to code completion but highly vulnerable to code infilling and translation.
• Strategy-type guardrails show mixed results: some methods (Self-Reminder, SmoothLLM) reduce ASR consistently, others degrade performance on particular base models; even the best strategy-type reductions left substantial residual ASR for FSA (largest reduction 27.7% but remaining ASR 64.7% in an example).
• Classifier-type guardrails perform well on straightforward text-to-code prompts (level 1 ASR 6.9%–27.5%) but degrade under obfuscation and stealthy FSA. FSA ASRs against input classifiers reached 73.3% and 98.5% for two evaluated systems. Output classifiers achieved variable F1s but also suffered on FSA.
• Multi-turn FSA remains effective: a pilot multi-turn study reported ASRs between 65.0% and 92.3% across guardrails.

Limitations

The benchmark is English-only, uses quantized versions of some LLMs so results may not generalise to full-precision models, and includes only a pilot multi-turn evaluation with limited samples. White-box guardrails that require internal neuron access or additional training were not evaluated. Dataset and labelling choices were constrained by budget.

Implications

Attackers can bypass many current defences by using jailbreak templates, obfuscation, or by embedding malicious requirements within plausible development scenarios (FSA). In practice adversaries could obtain working malicious code from LLMs via natural-language prompts, obfuscated inputs or by submitting partial code for infilling or translation. Multi-turn decomposition further increases success. These gaps enable scalable, low-cost malware production and make LLM-assisted software development an attractive attack surface for adversaries.