Stronger transfer attacks can also be the stealthier ones
Reviewing Pareto-Improving Adversarial Attacks with Primal-Dual Regularization · Yang Dai, Longfei Zhang, Wei Tao et al.
Researchers show that the supposed trade-off between how well an adversarial image transfers and how visible its noise is comes from comparing attacks at one fixed budget. A lightweight wrapper then makes strong transfer attacks look cleaner without losing success. Defences tested at a single budget face the weaker version of the threat.
The usual story about transfer attacks on image classifiers is that strength costs visibility. Push harder to fool a model you cannot see, and the noise on the image gets easier to spot. This research says that bargain was never real. It was a measurement habit.
A transfer attack is the practical black-box case. The attacker crafts an adversarial image against a surrogate model they own, such as ResNet-50 or a vision transformer. They then hope it fools a different target model they cannot inspect. Every attack works under a budget: a cap on how far any single pixel may move. Benchmarks usually fix that cap and compare success rates.
Under that protocol, stronger attacks fool more targets but look worse. The researchers point out that this compares attacks at a single snapshot. Sweep the budget instead, and plot success against image quality. The stronger attacks already come out ahead. At matched success they look cleaner, because they reach it with a smaller budget.
The fixed cap was hiding the advantage
Their wrapper is built to exploit that. Strong attacks tend to saturate. They drive many pixels right up to the maximum allowed change, and that produces the grainy, uniform texture that quality metrics pick up. The wrapper adds a penalty on that saturation. It solves the penalised problem with a paired update: one step on the perturbation, and one on a helper variable that carries the penalty. The base attack's gradient direction is left alone. There are no extra networks, no saliency maps and no hand-built models of human perception.
What I like is how little it asks. It slots into the update step of seven existing attacks and adds negligible runtime. At the common budget of 16/255, a learned measure of how different two images look improved by about 17 percent on average. Success was held or improved. At high success rates, the best wrapped attacks beat purpose-built stealth attacks that lean on saliency maps, perception priors or generative models.
The preconditions are the familiar ones for this class of attack. The attacker needs direct control of the pixels, so this is digital only; physical attacks were not tested. They need a surrogate model, and the gains vary with which one they pick. The penalty strength also needs tuning per attack. Stealth was scored with image-quality metrics rather than human viewers.
For defenders, the lesson sits in evaluation rather than in any single detector. If you benchmark a robust model or a perturbation detector against attacks at one fixed budget, you are testing against the weaker version of the threat. I would test across the sweep, against attacks tuned to look as clean as possible at the success rate they need.
Strength and stealth were never really opposed. The benchmark made them look that way.
Further analysis
Problem
The paper studies the tension between transferability and imperceptibility in black‑box adversarial attacks. Under the standard fixed perturbation budget protocol, stronger transferable attacks achieve higher attack success rate (ASR) but often produce more visible artefacts. The authors argue this apparent trade‑off is an evaluation artefact of fixed‑epsilon snapshots rather than an intrinsic conflict, and ask whether a modular optimisation wrapper can unlock the latent imperceptibility advantage of strong transfer attacks without sacrificing transferability.
Approach
They propose ST, a plug‑in primal‑dual wrapper that modifies only the projection/update step of an existing transfer attack. ST adds an L∞ saturation regulariser to the constrained attack objective and, via Fenchel duality, converts it into a saddle problem where the L∞ penalty appears as an L1 dual‑ball constraint. The optimiser then performs a two‑step primal‑dual update: a projected primal step on the perturbation plus an L1‑ball projection on a dual variable, with optional relaxation. ST preserves the base attack’s ascent direction (momentum, transformed gradients, MDCS variants) and requires no auxiliary networks or handcrafted perceptual priors. Experiments follow a transfer evaluation protocol on 1000 ImageNet‑compatible images, sweeping ε∈[2/255,32/255], using T=10 iterations and multiple surrogates (ResNet‑50 and ViT‑B/16) and seven target models spanning CNNs and ViTs.
Key Findings
- Pareto perspective: When comparing ASR versus imperceptibility across an ε sweep, stronger transfer attacks already weakly dominate weaker ones — fixed‑ε comparisons can be misleading.
- ST extends Pareto frontiers: Across seven representative base attacks, ST shifts each attack’s ASR–imperceptibility curve outward, yielding better perceptual metrics at matched ASR and higher ASR at matched perceptual quality.
- Quantitative gains at ε = 16/255: Average per‑attack improvements reported are ≈17% reduction in LPIPS and ≈14% improvement in NIQE while ASR is preserved or improved on most variants.
- Strong exemplar: STMI (MI‑FGSM wrapped with ST) reduced LPIPS by 31% and NIQE by 21%, increased MS‑SSIM by 3% and TOPIQ‑FR by 16% while maintaining competitive ASR.
- Competitor comparison: At matched high‑ASR regimes, top ST variants (e.g. STOPS, STMDCSMEF) Pareto‑dominate dedicated stealth‑oriented transfer methods that rely on saliency maps, JND priors or generative models.
- Runtime and modularity: The wrapper adds negligible runtime overhead relative to a surrogate forward/backward pass and is compatible with diverse attack direction pipelines.
Limitations
Evaluation relies on objective image‑quality metrics (full‑reference, no‑reference and distributional) rather than human perceptual studies; these metrics can disagree and do not fully substitute for user studies. Results are limited to digital, pixel‑bounded perturbations and may not transfer to physical attacks. Some gains vary with surrogate architecture and require per‑attack tuning of the regularisation scalar; FID behaved inconsistently in some cases.
Implications
Offensive implication: An attacker can wrap any strong transferable attack with ST to produce adversarial examples that are both highly transferable and substantially harder to detect by common perceptual metrics, without additional models or perceptual priors. This enables stealthier black‑box attacks at the same ASR, or higher ASR at the same perceptual quality, undermining detection strategies and fixed‑budget evaluation protocols that do not consider Pareto frontiers.
The paper title, authors and abstract
Pareto-Improving Adversarial Attacks with Primal-Dual Regularization
Transferable adversarial attacks are arguably the most practical black-box threat model. Under the same perturbation budget, stronger transfer attacks attain higher attack success rate (ASR), yet their imperceptibility also tends to degrade. Under such a fixed-budget protocol, transferability and imperceptibility therefore appear to trade off against each other. We argue that this conflict is an artifact of fixed-budget evaluation, not an intrinsic trade-off. When attacks are compared on the ASR--imperceptibility Pareto frontier obtained by sweeping $ε$, stronger transfer attacks already attain better imperceptibility at matched ASR than weaker ones. To exploit this latent advantage, we introduce the stealthy transfer attack ST, a plug-in primal-dual wrapper that adds an $L_\infty$ saturation regularizer to the standard constrained objective and resolves it through a two-step primal-dual update: a projected primal step on the perturbation coupled with an $L_1$-ball projection on a dual variable that absorbs the regularizer through Fenchel duality, requiring no auxiliary models or handcrafted perceptual priors. Empirically, ST extends the Pareto frontier across different base attacks and additional surrogate architectures. At $ε{=}16/255$, average imperceptibility gains over each base attack are $17\%$ on LPIPS and $14\%$ on NIQE while ASR is preserved or improved. At matched high-ASR levels, the strongest ST variants further Pareto-dominate dedicated stealth-oriented transfer attacks, confirming that the latent imperceptibility advantage of strong transfer attacks can be unlocked by a primal-dual optimization wrapper without sacrificing transferability. Code will be made available at \url{https://github.com/AndssY/ST}.
