Single Texture Steers Multitask Robot Agents Off Course

Problem

This paper studies whether a single adversarial surface appearance can systematically influence multitask Vision-Language-Action (VLA) policies across many manipulation tasks and viewpoints, and whether such a single texture can produce targeted changes in predicted robot actions. This matters because VLAs directly control embodied agents, so adversarial visual patterns that transfer across tasks could cause unsafe motions, failed manipulation, or physical damage in deployed robotics systems.

Approach

UniTexture creates a single object-bound UV texture and optimises it jointly across a distribution of tasks, instructions, states and camera viewpoints. A differentiable renderer is first calibrated on multi-view clean observations so that rendered composites match simulator appearance, then rendering parameters are frozen. During optimisation, the texture is composited into sampled task frames and gradients are backpropagated from the frozen VLA policy outputs to the texture only. The targeted loss is defined in each model’s native action representation, supporting both discrete autoregressive action tokens and continuous flow-matching action chunks. Evaluations use two VLA checkpoints (OpenVLA and π0.5) on the LIBERO-Spatial and LIBERO-Goal suites; controls include renderer-free clean observations, rendered original textures and a fixed Gaussian-noise texture.

Key Findings

• Aggregate task disruption: across evaluated settings UniTexture reduces mean task success rate from 90.0% under benign conditions to 48.4% under attack, demonstrating broad cross-task vulnerability.
• Targeted action steering: for the flow-matching model (π0.5) UniTexture produces substantial targeted direction shifts, with Target Direction Shift (TDS) of 10.6 and 17.9 and Paired Direction Hit Rates (pDHR) of 61.0% and 65.5% for two object targets, while task success falls to 33% and 40% on those settings.
• Model-dependent disruption patterns: OpenVLA is strongly disrupted (for example Spatial: clean ~84% down to 53%/25% for two object textures) but often shows weaker or even negative average signed directional shifts, indicating disruption can arise from destabilising visual–action mappings rather than coherent steering.
• Cross-suite and cross-model transfer: textures optimised on one task suite or model retain effectiveness without re-optimisation. Transfer is asymmetric: textures from π0.5 often reduce OpenVLA success substantially (to 26%–61%), whereas textures from OpenVLA typically leave π0.5 success high (92%–97%).
• Directional asymmetry and geometric dependence: upward motion targets (+z) are generally most effective at inducing failure, while downward targets are least effective, reflecting interaction geometry.

Limitations

The evaluation is performed in simulation on LIBERO suites and on two specific VLA checkpoints, so real-world sensor noise and unmodelled dynamics are not explored. The attack assumes the adversary can repaint a single object and that gradients from the frozen policy are available during offline optimisation. The renderer is calibrated from the original asset and kept fixed, so effectiveness may depend on calibration quality and mesh fidelity.

Implications

An attacker who can persistently alter an object’s appearance can steer or destabilise a multitask VLA across many instructions and scenes, causing unsafe or undesired physical behaviour. The attack’s cross-task and cross-model transfer without per-task optimisation makes it practical for deployment-level adversaries, increasing the threat that a single manipulated prop could be reused to affect multiple robotic tasks and platforms.