Small patch derails flow-matching robot control

Problem

This paper examines whether flow-matching vision-language-action (VLA) policies that generate actions by integrating a learned denoising velocity field are truly robust to input perturbations. Previous attacks targeted final actions or embeddings and reported apparent robustness; the authors ask whether that robustness hides a vulnerability in the multi-step denoising ordinary differential equation that these models integrate at inference, and whether a small physical patch can reliably derail a deployed, frozen policy.

Approach

The authors propose DRIFT, a test-time, universal adversarial patch optimised offline with white-box gradients and then applied unchanged to a wrist-camera view over the robot gripper. Rather than attacking final actions or encoder embeddings, DRIFT directly maximises divergence in the model's denoising velocity field. The threat model is untargeted and conservative: the attacker may not modify weights or training data, places a single physical sticker (default 32×32 pixels, about 2% of the wrist image) over the gripper, and optimises the patch with projected gradient descent for 500 iterations using a pool of wrist observations. Victim policies are the flow-matching models pi0 and pi0.5 with K=10 Euler denoising steps, evaluated on four LIBERO suites. Metrics include attack success rate (ASR) defined relative to clean task success, and Normalised Action Discrepancy (NAD).

Key Findings

• Single-step vulnerability: perturbing only the first denoising step (k=0) achieves markedly higher velocity divergence and ASR than attacking later steps; early-step attacks (k ≤ 5) break essentially all tasks whereas late-step attacks fail.
• Fewer steps, stronger attack: attacking only the first step outperforms attacking a wider early-window. Adding steps weakens the attack due to gradient conflict among per-step input-space gradients, which are misaligned and partially cancel when summed.
• High effectiveness and efficiency: DRIFT breaks nearly all originally-solvable tasks on pi0 across all four LIBERO suites (near 100% ASR) and far exceeds action-space (UADA ~13.2% ASR) and embedding-space (EDPA ~25.3% ASR under larger patch) baselines. The single-step objective reduces optimisation cost by roughly a factor of K compared with full-trajectory attacks.
• Failure mode and mechanism: patched rollouts commonly produce a "phantom grasp" where the gripper closes immediately and the arm never approaches the object. Attention analysis shows the patch absorbs a large fraction of the object token's attention (28-33% vs 1.7% under clean input), indicating misdirected internal focus.
• Constraints on realism and transfer: a grayscale-constrained patch can be stealthy but needs about twice the perturbation budget; attack success shows a sharp threshold in l-infinity budget (e.g. ASR jumps near epsilon=0.1). Transfer across model versions is weak, indicating model-specific patches and the need for white-box access during optimisation.

Limitations

The attack requires offline white-box access to the frozen policy for gradient-based optimisation and produces a model-specific universal patch; cross-model transfer is limited. Some mitigations are partially effective but not universal: JPEG compression can largely block DRIFT on simpler suites once quality is reduced sufficiently, but long-horizon tasks remain partly vulnerable. The study focuses on untargeted failures and on the specific flow-matching architectures evaluated.

Implications

An attacker with temporary white-box access to a deployed flow-matching VLA can craft a small, physically realisable sticker that reliably induces untargeted task failure by perturbing only the first denoising step. The vulnerability arises from the denoising ODE's cascade amplification and an input-space gradient conflict that makes single-step perturbation both more effective and cheaper to compute than multi-step attacks. This demonstrates a practical, real-world attack surface on continuous denoising-based control that goes beyond previous action- or embedding-space threats.