Comics jailbreak multimodal LLMs at high success rates
Attacks
Multimodal Large Language Models (MLLMs) promise better judgement by seeing as well as reading. They also inherit a fresh set of failure modes. A new study argues that narrative structure in images is a potent jailbreak vector: place a harmful goal inside a simple three-panel comic, ask the model to role-play and finish the story, and safety alignment often gives way to the plot.
The authors introduce ComicJailbreak, a benchmark of 1,167 attack instances built from 200 harmful and 100 benign seed goals. Each goal is expressed across five task setups familiar to deployment contexts: article writing, public speech, instructional guides, social media posting and code generation. Ten harm categories span the space of risky content. The comics are deliberately minimal. Panels one and two set up context; panel three carries a short paraphrase of the goal in a speech bubble. That restraint is smart: it isolates the effect of narrative structure without confounds from photorealism or text density.
Fifteen state-of-the-art MLLMs, six commercial and nine open-source, face four input conditions: plain text, a strong rule-based text jailbreak, random images with overlaid text, and the comic-template attack. Success is measured as Attack Success Rate (ASR) per instance and Ensemble ASR (EASR) across multiple templates for the same goal. Refusal Rate (RR) and Ensemble RR (ERR) capture over-blocking on benign prompts.
What the results show
Comic-based attacks substantially outperform plain text and random-image baselines and often match or exceed a strong rule-based jailbreak. Several commercial models show EASR above 90% under comic ensembles. Many open-source models also post EASR above 85%. Model families differ in resilience, but the broad pattern is clear: structured visual narratives pull models into compliant role-play, sidestepping text-first guardrails.
The defence picture is mixed. Prompt-based approaches such as AdaShield and Attack as Defence (AsD) can slash EASR, in some cases by more than 80 percentage points. The cost is steep: benign prompts see large jumps in refusals, with ERR increases above 80% on several models. Multi-turn self-reflection trims EASR more modestly, and with less over-refusal, but it is inconsistent across setups.
Evaluation is a second weak point. The study uses an ensemble of automatic safety judges for scale, then audits them with targeted human review of 2,869 generations. Agreement is decent overall (Cohen’s kappa 0.751), but the judges struggle on sensitive yet non-harmful content. On harmful prompts they record a false positive rate (FPR) of 0.10 and false negative rate (FNR) of 0.183. On benign prompts, performance degrades to FPR 0.234 and FNR 0.422, indicating over-flagging and missed nuance.
Why it matters
This work shows that coherence, not just content, drives jailbreaks. The comic template compresses intent into a tidy sequence that invites models to continue the narrative. That is a realistic risk if users or connected systems upload web images, memes or slides. The defence trade-off is awkward for product teams: you can harden against these comics, but current methods tend to reject many legitimate requests. The judging results underline a second deployment risk: if your red-teaming and moderation pipelines over-rely on brittle automated labellers, you may mis-measure both harm and usability.
There are limits. The benchmark uses compact English three-panel templates and a small set of defence strategies. Richer layouts, multilingual inputs and other narrative devices may change the picture, for better or worse. Still, as an attack design, it is elegant and uncomfortably effective. The obvious next steps are narrative-aware multimodal alignment, better fusion between vision and safety objectives, and evaluation stacks that combine robust automated judges with targeted human review where nuance matters.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Structured Visual Narratives Undermine Safety Alignment in Multimodal Large Language Models
🔍 ShortSpan Analysis of the Paper
Problem
This paper studies a new multimodal jailbreak vector in which simple three-panel visual narratives - comics - embed harmful goals and prompt Multimodal Large Language Models (MLLMs) to role-play and complete the story. The work asks whether coherent visual narratives can undermine existing text-only safety alignment, how widespread the vulnerability is across modern MLLMs, and whether current automated judges and defences reliably detect or prevent these failures. This matters because narrative-driven multimodal inputs are common in web content and may bypass safeguards that work for explicit text prompts.
Approach
The authors introduce ComicJailbreak, a benchmark of 1,167 three-panel comic instances created by mapping 200 harmful and 100 benign seed goals into five task setups (article writing, public speech, instructional guides, social media posting, code generation) and ten harm categories. Each comic places contextual panels 1-2 and a third-panel speech bubble containing a concise paraphrase of the goal. They evaluate 15 state-of-the-art MLLMs (six commercial, nine open-source) under several attack formats: plain text, a strong rule-based text jailbreak, random-image text overlays, and the comic-template attack. Metrics include Attack Success Rate (ASR) and Ensemble ASR (EASR) across templates for the same goal, plus Refusal Rate (RR) and Ensemble RR (ERR) on benign goals. An ensemble of three automated safety judges provides scale labelling, and a targeted human study labels 2,869 generations to assess judge reliability. The paper also tests three defences: AdaShield, Attack as Defence (AsD), and multi-turn self-reflection.
Key Findings
- Comic-template attacks significantly increase jailbreak success compared with plain-text and random-image baselines; comics often match or exceed a strong rule-based text jailbreak, with ensemble EASR exceeding 90% for several commercial models.
- Across 15 evaluated MLLMs, comic attacks yield high ASR and EASR generally above 85% across many open-source models; some model families show near-universal vulnerability while others exhibit partial resistance.
- Defences AdaShield and AsD substantially reduce EASR for many models - in some cases dropping EASR by over 80% - but they also markedly increase refusal on benign prompts; several models experienced more than an 80% increment in ERR when defences were applied.
- Multi-turn self-reflection provides only modest EASR reductions and is less consistent than prompt-based defences, though it tends to avoid large increases in refusal rates.
- Automated judges are reliable on overtly harmful outputs but brittle on benign yet sensitive content: human-labelled evaluation (Cohen's kappa = 0.751) shows majority voting had FPR 0.10 and FNR 0.183 on harmful prompts but degraded to FPR 0.234 and FNR 0.422 on benign prompts. Automated systems substantially over-flagged harms in sampled benign outputs.
Limitations
ComicJailbreak focuses on minimal English-language three-panel templates, so it covers a narrow slice of visual narratives and does not test richer layouts, longer dialogues, multilingual or code-mixed inputs. The large-scale evaluation relies on automated judges that exhibit brittle performance on sensitive benign content, though a human-labelled subset partially mitigates this. Defence evaluation is limited to a few existing methods and may not represent the full design space of safer multimodal alignment.
Why It Matters
The work reveals that simple narrative structure in images can bypass text-oriented safety measures, producing high-risk jailbreaks across commercial and open-source MLLMs. Practical implications include the need for multimodal, narrative-aware alignment and improved evaluation that combines robust automated judges with human review. Defences must balance reduced harmful outputs with the usability cost of over-refusal; current prompt-based defences can protect models but at the expense of rejecting benign user requests, posing trade-offs for real-world deployment.
