Researchers Expose Transferable Black-Box Prompt Injection

Problem

Direct Prompt Injection DPI attacks pose a critical security threat to large language models due to their low barrier of execution and potential for substantial damage. Existing white box and gray box methods are often impractical and black box approaches suffer from limited transferability. The paper investigates a transferable, gradient free black box attack that can exploit LLM prompts across multiple models and unseen tasks, highlighting the difficulty of defending prompt based threats that operate outside model internals.

Approach

The authors propose an activations guided prompt injection framework. They build an energy based model EB M from activations of a surrogate model to evaluate the quality of adversarial prompts. Guided by the EBM, they apply token level Markov Chain Monte Carlo sampling to adaptively optimise prompts, enabling gradient free black box attacks. The method begins with a template dataset of prompts, decomposed into prefix infix and suffix components. Data augmentation decouples these components to maximise diversity, creating a template set of prefixes infixes and suffixes. Activations are collected from a surrogate model during prompt evaluation, constructing an activation dataset used to train the EBM. A two layer multilayer perceptron serves as the activation classifier to realise the energy function. For generation, a seed prompt is iteratively mutated: a token is replaced using a masked language model, the activations are recomputed, and the energy is used to accept or reject the new candidate. The process continues for a fixed number of iterations, yielding an adversarial prompt with high attack effectiveness. The Tensor Trust Attack dataset supplies attack samples, and several tasks are used for training and testing to evaluate transferability. Open source and closed source victim models include Qwen and Llama variants and GPT 4o mini. Evaluation uses Attack Success Rate and Transfer ASR, with baselines including human crafted prompts and prior white and gray box methods. The study also analyzes naturalness via perplexity and interprets activation based patterns.

Key Findings

• High cross model transferability: the method achieves 49.6 per cent attack success across five mainstream LLMs, outperforming human crafted prompts by about 34.6 per cent.
• Robust unseen task performance: the approach maintains 36.6 per cent ASR on tasks not encountered during training.
• Activation driven interpretability: a strong negative correlation exists between energy and ASR, with a Pearson correlation of about 0.979, indicating lower energy aligns with more effective prompts; PCA of activations shows successful attacks move toward specific activation directions.
• Prompts remain natural: perplexity metrics show generated prompts maintain naturalness similar to manual prompts and other adversarial methods, avoiding overtly unnatural text that could trigger detection.
• Hierarchical prompt structure and dataset design: using 85 prefixes, 35 infixes and 85 suffixes, combined into thousands of templates, enabling diverse, realistic prompts while preserving attack strength.
• Defence implications: results suggest the need for robust prompt validation, monitoring of activation patterns and cross model testing to improve resilience against transferable prompt based attacks.

Limitations

The work notes a trade off between naturalness of prompts and attack strength, with naturalness not explored beyond current acceptability thresholds. The method does not address defenses based on text classifiers, and performance varies with task and model choices. Further work could explore sharper constraints to maintain naturalness while increasing attack potency and evaluating against more defensive mechanisms.

Why It Matters

The study demonstrates a transferable black box direct prompt injection attack that leverages activation signals from a surrogate model and token level MCMC optimisation, illustrating a practical vulnerability that transfers across models and tasks. It highlights blind spots in defenses that focus on model internals, underscoring the need for robust input validation, monitoring of activation patterns, and adversarial prompt testing across models to improve resilience. The findings have security and trust implications for LLM powered services used in critical applications, reinforcing the importance of comprehensive defensive frameworks and regular red teaming across model families.