Partial audio deepfakes evade humans and detectors

Problem

The paper examines how advances in speech synthesis affect human and automated detection of deepfake audio. Earlier studies reported high human detection accuracy but mainly evaluated older synthesizers. The authors test whether modern commercial synthesis and a realistic attack variant called partial spoofing, where a single sentence in an otherwise genuine recording is replaced, undermine human judgement and existing automated detectors. This matters for fraud, disinformation and voice‑based social engineering because short, high‑quality inserts can change meaning while appearing credible.

Approach

The authors ran a controlled questionnaire study with 82 IT professionals listening to recordings for nine well‑known English‑speaking public figures. Three synthesis systems were selected to represent 2019, 2022 and 2024: Real‑Time Voice Cloning (RTVC), YourTTS, and the commercial ElevenLabs. For each speaker participants judged seven recordings composed of four sentence‑level segments: one bona fide, three full‑spoof (all sentences synthetic) and three partial‑spoof (one sentence replaced by a synthetic sentence). Replacement text was generated with ChatGPT‑4 and normalized in post‑processing. Six pretrained automated detectors (two front‑ends, two back‑ends, trained on ASVspoof datasets) were run on the same material. Evaluation used F1, a strict All OK metric (all four segments correct), and Over 50 Percent, with additional analyses of sensitivity (d′), response bias, and inter‑rater agreement.

Key Findings

• Detection of full‑spoof audio remains high for older tools: RTVC and YourTTS yielded human F1 scores around 0.90 and near‑unanimous per‑sentence accuracy.
• ElevenLabs full spoofs were far harder: mean human F1 dropped to 0.48, per‑sentence accuracy fell to about 0.43 (near chance), and strict All OK success was 28% despite participants being warned that deepfakes might appear.
• Partial spoofing is especially deceptive: when a single sentence was inserted from ElevenLabs, mean human F1 fell to 0.29 and All OK rate dropped below 10%; the synthetic sentence was judged bona fide 77% of the time (detected only ~22.65%).
• Humans and detectors fail in complementary ways: detectors perform well on older generators but collapse on out‑of‑distribution ElevenLabs full spoofs (mean detector F1 ~0.09 versus human 0.48); on partial ElevenLabs detectors show higher whole‑recording F1 (~0.40) but their strict All OK localisation rate is 0.00, so neither humans nor detectors reliably localise short manipulations.
• Listeners increasingly misclassify genuine audio as fake, with a reported false‑alarm rate of about 17.5%, indicating erosion of trust in bona fide speech.

Limitations

The sample comprises IT professionals and may not generalise to the wider public. Participants were explicitly primed that deepfakes could be present and the experiment used a high fixed prevalence of synthetic content, which can raise suspicion and false positives. Only one model represents each release period and ElevenLabs is the sole commercial system, so results reflect system‑specific differences rather than a general temporal trend. Uncertain responses were excluded and the dataset is not public due to identifiable voices.

Implications

Offensively, modern commercial speech synthesis can produce short inserts that are frequently accepted as genuine, enabling convincing targeted misinformation, fabricated evidence and voice‑phishing attacks that change the meaning of otherwise bona fide messages. Attackers can exploit segment‑level edits to evade both human scrutiny and whole‑recording detectors. The complementary failure modes of humans and detectors mean a single defence is insufficient; adversaries benefit from out‑of‑distribution generators and contextually embedded edits. These results argue that attackers can scale plausible social‑engineering campaigns and that defenders should prioritise provenance, watermarking, procedural verification and segment‑localisation methods over reliance on human perception alone.