New to ShortSpan? We distil the AI-security research that matters into practitioner takeaways — edited by Ben Williams (NCC Group).Get the weekly email
// Analysis
Researchers Expose AI-Driven Phishing Risks at Scale
Published: Mon, 01 Sep 2025 • By Marcus Halden
Attacks
A new systematization shows how large language models rapidly enable scalable, convincing phishing campaigns. The study categorizes generation methods, attack features, and defenses, finding mass-produced credible messages, patchy detection, and scarce public datasets. Organizations face higher fraud risk and need layered defenses plus stronger, realistic testing now.
AI is turning phishing from a craft into a factory. A new systematization of research maps how large language models let attackers mass-produce convincing phishing messages in minutes, and why that matters for organizations and everyday users.
The paper introduces GenCharDef, a practical framework that walks through how phishes are generated, what makes them effective, and how we try to stop them. Key findings include clear evidence that attackers can scale campaigns quickly, often achieve worrying click rates with AI-crafted spear phishing, and increasingly run Phishing-as-a-Service operations. Detection tools are improving but vary widely depending on prompt style, domain, and the data defenders can access.
Why this is worrying: defenses rely on brittle signals and private datasets. Many research corpora are not publicly available, model updates change attacker behavior, and common evaluation metrics miss how realistic or evasive an attack is. In plain terms, defenders may be testing against last year’s threats while attackers iterate in real time.
Practical limits of the study include dependence on available papers through 2025 and uneven public data. Still, the writeup gives a useful roadmap for defense in depth: combine message analysis, URL and metadata checks, user training, and red-team style realistic testing. A little humor helps here: treat your inbox like a suspicious guest at a party and verify before you offer the keys.
Operational takeaways
Assume attackers can generate realistic messages fast; increase automated filtering.
Use layered checks: content, links, sender metadata, and behavioral signals.
Prioritize open, realistic datasets and continuous red-team testing.
Train users with up-to-date, example-based simulations rather than static warnings.
Additional analysis of the original ArXiv paper
📋Original Paper Title and Abstract
SoK: Large Language Model-Generated Textual Phishing Campaigns
End-to-End Analysis of Generation, Characteristics, and Detection
Authors: Fengchao Chen, Tingmin Wu, Van Nguyen, and Carsten Rudolph
Phishing is a pervasive form of social engineering in which attackers
impersonate trusted entities to steal information or induce harmful actions.
Text-based phishing dominates for its low cost, scalability, and
concealability, advantages recently amplified by large language models (LLMs)
that enable ``Phishing-as-a-Service'' attacks at scale within minutes. Despite
the growing research into LLM-facilitated phishing attacks, consolidated
systematic research on the phishing attack life cycle remains scarce. In this
work, we present the first systematization of knowledge (SoK) on LLM-generated
phishing, offering an end-to-end analysis that spans generation techniques,
attack features, and mitigation strategies. We introduce
Generation-Characterization-Defense (GenCharDef), which systematizes the ways
in which LLM-generated phishing differs from traditional phishing across
methodologies, security perspectives, data dependencies, and evaluation
practices. This framework highlights unique challenges of LLM-driven phishing,
providing a coherent foundation for understanding the evolving threat landscape
and guiding the design of more resilient defenses.
🔍ShortSpan Analysis of the Paper
Authors
not reported
Problem
The paper provides the first systematisation of knowledge on large language model generated phishing, offering an end to end analysis of generation techniques, attack features and mitigation strategies. It introduces the Generation Characterisation Defence GenCharDef framework to compare LLM phishing with traditional phishing across methodologies, security perspectives, data dependencies and evaluation practices, highlighting unique challenges and guiding the design of more resilient defence.
Approach
The study delivers an end to end SoK on text based phishing driven by LLMs, based on a meticulous literature review of forty four studies. It proposes GenCharDef as a multi stage taxonomy that covers generation, characterisation and defence, and synthesises findings on datasets, evaluation practices, attack vectors and mitigation strategies. It also identifies open challenges and opportunities and outlines ethical commitments including release of non sensitive artefacts.
Key Findings
GenCharDef provides a structured view of how LLMs enable phishing generation, how it differs from traditional phishing and how current defences respond across data dependencies and evaluation practices.
LLMs support scalable phishing operations including Phishing as a Service, with high engagement and rapid content production; evidence shows notable click through rates for LLM generated spear phishing and widespread use in campaigns in 2025.
Detection and defence trends are moving toward intent level analysis and multi agent semantics that integrate content with URLs and metadata; prompt screening and semantic embeddings are increasingly used, but performance varies by prompt and domain.
Dataset availability is limited with around sixty percent private or non reproducible corpora and few open sources; model version changes complicate reproducibility over time.
Current evaluation relies mainly on accuracy and standard classification metrics; there is a need for richer benchmarks that capture realism, manipulation strength, contextual relevance and evasion resilience, with growing attention to evasion oriented metrics in some work.
Limitations
Main constraints include reliance on existing literature up to twenty twenty five, substantial use of private datasets which hinders reproducibility, varied evaluation metrics with limited standardisation and challenges in cross domain cross language transfer and attacker intent migration.
Why It Matters
The work documents societal security risks from AI driven phishing at scale, including manipulation and fraud risks affecting trust privacy and critical sectors; it provides a foundation for defence in depth guiding detection strategies, evaluation practices and the design of more robust defences against evolving phishing threats.
- LLM-powered phishing can produce much more believable and evasive emails, often slipping past common detectors.
- The study shows detectors (older ...
- It presents a scalable framework that simulates attacker-adaptive phishing using LLMs and genetic algorithms, showing how attacks can become more pe...
- It shows a big gap between “recognizing phishing” and “refusing to generate phishing,” especially in offline/open-source LLMs, highlighting a practi...
- Shows that LLM-based phishing detectors can be subtly manipulated by prompt injection, potentially letting fake sites slip through.
- Introduces a ...
- Relevance: Examines how AI-generated phishing (including adversarially perturbed emails) can evade detection and offers a robust detection pipeline ...
- Why it’s relevant: It shows how LLM-based phishing defenses can be both effective (over 90% detection in tests) and fragile to adversarial tactics l...
- It exposes how AI can be steered to produce highly convincing spear-phishing content through jailbreak prompts and a multi-agent critique loop, illu...
