Adversarial Noise Hijacks Speech Enhancement Outputs

Problem

Are modern speech enhancement SE systems vulnerable to adversarial attacks? The paper investigates how increasingly expressive predictive and generative SE models can be manipulated by adversarial noise that is psychoacoustically masked by the input, causing the enhanced output to convey a different semantic meaning. The authors experimentally verify vulnerabilities in contemporary SE models and note that diffusion models with stochastic samplers show inherent robustness by design. The findings have security relevance for audio pipelines in telecoms, voice assistants and transcription, with implications for covert commands, impersonation, misinformation and broader surveillance and manipulation concerns.

Approach

The study conducts targeted white box attacks against three SE variants using a psychoacoustic masking constraint and an optimisation based on an l2 budget. The SE models include direct predictive mapping, a complex ratio mask predictive model and a diffusion based score based model using a Noise Conditional Score Network backbone. The diffusion model is evaluated with two inference modes: a standard stochastic reverse diffusion and a fixed noise path to remove sampling variability. Attacks operate on the complex short time Fourier transform domain, adding a perturbation to the source mixture to steer the enhanced output towards a chosen attacker target. The optimisation uses gradient descent with momentum and a psychoacoustic gate derived from the MPEG-1 masking model to keep perturbations inaudible. They employ activation checkpointing to manage memory when backpropagating through the diffusion chain. Experiments use the EARS WHAM v2 dataset for training and evaluation on 100 random pairs drawn from 850 utterances, with a hidden target attacker utterance per pair.

Key Findings

• Direct Map predictive attacks can be almost perfectly guided to the target, but may be audible if unconstrained; under a fixed energy budget and psychoacoustic masking, the perturbation remains louder but the attacker still achieves high target alignment with a noticeable trade off in output quality.
• Complex Ratio Mask predictive attacks show similar vulnerability to Direct Map under comparable constraints, with perturbation impact and target matching closely aligned though sometimes marginally less successful.
• Diffusion based SE is more robust to targeted manipulation. Without constraints it can be steered but at higher audible cost and weaker target match; with the same energy budget it achieves poorer target alignment yet remains more audible. Freezing the random path during attacks increases success, whereas more reverse steps or larger diffusion scales can hinder control. Increasing the maximum diffusion scale and stochastic sampling generally improves robustness, whereas reducing it reduces robustness.
• Overall, the study finds that stochastic sampling in reverse diffusion moderately raises resilience to adversarial manipulation, indicating concrete defence directions and design trade offs for building more secure speech processing systems.

Limitations

Limitations include the white box experimental setting, reliance on the EARS WHAM v2 dataset, computational demands addressed via checkpointing, and the focus on only two families of SE architectures. Generalisability to other datasets or real world telecom environments remains to be tested.

Why It Matters

The work highlights security risks in speech processing pipelines, including covert commands, impersonation and dissemination of misinformation mediated through manipulated audio. The observed robustness of diffusion SE under adversarial manipulation points to defensible design choices and trade offs between robustness, quality and latency. The study also provides a methodological framework for evaluating adversarial vulnerability in SE systems to inform mitigations and best practices in the cybersecurity community.