Diffusion Patches Steer VLA Robots in Black Box

Problem

This paper studies the vulnerability of Vision-Language-Action (VLA) models to visually natural, physically placeable adversarial patches. VLA systems map camera observations and language instructions directly to robot actions and are increasingly deployed for general-purpose manipulation. Prior attacks often produce conspicuous artefacts, require white-box access, or are inefficient, limiting real-world deployability. The work asks whether an attacker can produce inconspicuous patches that reliably steer robot behaviour in both white-box and realistic black-box scenarios.

Approach

The authors propose DURA, a diffusion-guided unrestricted patch attack that searches in the latent trajectory of a pretrained diffusion model to generate natural-looking patch content while optimising a targeted action objective. A single patch is optimised across diverse frames and instructions to improve transferability. In the white-box setting DURA uses gradients through the policy and decoder; in the black-box setting it estimates attack directions from action-output queries via a score-function estimator. Optimisation operates in the VAE latent space with a clean-anchor regulariser to keep patches visually coherent. Evaluations use OpenVLA and π0-FAST models on simulation and real-world benchmarks including LIBERO and BridgeData V2, and deployment tests on a Franka 7-DoF arm observed by a single fixed RGB camera.

Key Findings

• DURA attains very high attack success rates. Reported ASR ranges from 79.3% to 100% across evaluated white-box and black-box settings; white-box ASR reached 100% on evaluated suites. For OpenVLA, black-box ASR was 86.0% with simulated patches and 79.3% with printed physical patches, substantially outperforming a NES-adapted baseline by over 40 percentage points.
• Targeted control and precision. DURA raises attack precision (consistency with attacker-specified actions) to an average of 82.0% in white-box and 67.3% in black-box settings, versus 25.0% for clean policies, indicating induced behaviours align with intended targets rather than incidental failures.
• Real-world transfer and controllability. Printed patches on a Franka arm reproducibly induced attacker-specified failure modes including action freezing and erroneous movements; inserting or removing the patch toggled behaviour, demonstrating on-demand control.
• Visual stealth and robustness. Diffusion-guided patches appear more coherent and closer to clean imagery across multiple perceptual and local-artifact metrics, and remain effective under input-transform defences such as JPEG compression, bit-depth reduction and Gaussian noise (for example ASR stayed around 90% under strong JPEG compression on one model and 100% on the other).
• Efficiency and query scaling. DURA achieves high ASR substantially faster than pixel-space baselines; in black-box trials ASR improves with per-update query budget K from 49% at K=128 to 100% at K=2048, while moderate budgets such as K=512 already yield high success.

Limitations

The study focuses on targeted single-patch attacks visible to a fixed RGB camera and evaluates two open-source VLA models on specific datasets and a Franka robot. Black-box optimisation incurs a nontrivial query cost during patch synthesis and benefits from batch optimisation over multiple frames. Effectiveness depends on patch size and placement; performance rises sharply past a small area threshold and stabilises at larger sizes. Results are reported for the chosen diffusion prior, seeds and optimisation hyperparameters, which may affect transfer to other settings.

Implications

An attacker can craft and physically place a visually ordinary patch in a workspace to covertly steer or freeze VLA-controlled robots, requiring only the printed patch at deployment. With diffusion guidance the patch can appear semantically plausible and evade simple visual inspection and common input-transform defences. Black-box synthesis requires action-output queries during preparation but not model internals at deployment, enabling attacks on closed-source systems. Such attacks expand the physical attack surface of embodied AI and can be used to induce specific unsafe behaviours or create an on-demand control switch over robot execution.