Structural jailbreaks beat language gaps, hit finance hardest

Problem

The paper studies whether two independent safety weaknesses in aligned large language models compound: a structural jailbreak class called Involuntary In-Context Learning (IICL), where a harmful request is framed as the final cell of a data-labeling task so the model completes it by pattern rather than judging content, and the multilingual safety gap, where alignment erodes outside English. Determining whether these weaknesses generalise across providers and languages matters because defenders must know if combined attacks amplify risk, especially for high-value domains such as financial abuse.

Approach

The authors apply a deterministic IICL operator and a StrongREJECT-style rubric judge to red-team two Google models - gemini-2.5-flash and gemini-2.5-flash-lite - on two 30-item benchmarks: HarmBench (general-harm behaviours) and FinProof (financial-abuse behaviours). Each behaviour was probed under a bare single-shot English baseline and under IICL constrained to produce output in one of four languages: English, Spanish, Hindi, Arabic. A single LLM judge graded responses for non-refusal, topicality and specificity/actionability. To check bias, an independent non-Google judge re-graded a replication. Output-language verification used Unicode script and language identification. Sample size was n=30 per cell, single seed; unreachable probes were excluded.

Key Findings

• IICL generalises across providers and is highly effective in English: attack success rose from a single-shot floor of up to 6.7% to 80–90% on HarmBench and to 97–100% on FinProof, indicating much greater potency than earlier English-only results on a different provider.
• The financial domain is especially vulnerable: FinProof responses under English IICL saturated near 100%, making financial-abuse behaviours the most exposed class tested.
• Contrary to the compounding hypothesis, forcing IICL outputs into non-English languages tended to reduce attack success rather than increase it: eleven of twelve non-English IICL conditions scored below their English baselines (sign test, p ≈ 0.003).
• The attenuation increases with lower-resource languages: on the stronger model the FinProof Arabic condition collapsed from 100% to 33%, while Spanish and Hindi also showed large drops.
• The primary mechanism for reduction is a relevance curse: once structural compliance is achieved, outputs in lower-resource languages are lower-quality, often partial or vague, and so fail the substance-grading rubric even if produced.
• Results replicate under an independent non-Google judge (Cohen's kappa = 0.86 on 377 paired verdicts), and 76.6% of non-English responses were verified as genuinely in the requested language, so the pattern is robust to judging and language checks.

Limitations

This is a pilot study with n=30 per cell and a single random seed; both targets are Google models so transfer is shown to one additional provider but not universally; only two lower-resource languages were tested; the judge was an LLM with a replication by another LLM but human grading remains open; the IICL operator used a five-benign-shot scaffold, so reported English rates are a lower bound on the operator's ceiling.

Implications

For an attacker the results show a concrete, portable structural jailbreak that is highly effective in English and particularly so for financial-abuse prompts; translating the same attack into lower-resource languages is generally not advantageous because it produces lower-quality harmful outputs or triggers English deflection. Offensive value therefore centres on the English structural surface; multilingual variants are less reliable as an exploitation strategy.