On-device LLMs enable stealthy living-off-the-land attacks
Attacks
The paper examines a plausible near-future threat: Large Language Models (LLMs) running on devices acting as components in living-off-the-land attacks. Living-off-the-land refers to abuse of legitimate tools and processes already present on a system to avoid detection. The researchers demonstrate a proof of concept where a local LLM, discovered among cached models and inference runtimes, is used to generate and refine code that performs persistence and covert file manipulation without calling external APIs.
This matters because the LLM becomes an automation engine for an attacker who already has some foothold. Rather than delivering a standalone malware binary, an adversary can probe the host for inference runtimes, choose a usable model, apply jailbreak-style prompts to bypass safety, and instruct the model to compose scripts that use normal system utilities. That sequence raises three practical risks for operations teams: higher stealth because actions look like normal tooling; faster attacker iteration because the model synthesises payloads on demand; and persistence that survives network isolations since the chain can run fully offline.
Breakdown of infrastructure touchpoints: local inference processes and runtimes act like model endpoints; GPUs and local accelerators are high-value resources and can host large workloads; vector stores and cached model files create new data paths and potential exfiltration targets; code-generation via local tooling threatens secrets in developer environments and system keys. Each touchpoint maps to traditional attack surfaces but with different detection characteristics because model-driven activity often blends with benign automation or developer workflows.
Diagram in words: attacker foothold -> discover local inference runtime -> select capable model -> feed jailbreak prompt -> generate code -> execute via normal system utilities -> create persistence. That chain is useful to think through during incident response because observable signals shift from unusual binaries and command-and-control traffic to local process spawning, file writes in typical paths, scheduled jobs and subtle changes to startup configuration.
Mitigations for SREs and security teams
Practical defence is defence in depth. First, inventory and reduce surface: catalogue inference runtimes, model caches, GPU access and vector databases and remove or harden any you do not need. Second, enforce least privilege: run inference services under dedicated unprivileged accounts, confine them with OS-level sandboxing and disallow direct access to package managers, startup locations and secret stores. Third, apply prompt and output controls: log prompts and model outputs where policy allows, and apply rudimentary output sanitisation to block instructions that call local toolchains or write to sensitive paths.
Operational controls include detecting behaviour rather than relying on signatures. Monitor for anomalous local code generation, unexpected interpreter launches, rapid model-related disk activity and changes to scheduled tasks. Treat model-driven activity as an indicator of attack (IOA) and bake corresponding alerts into incident playbooks. On the supply chain side, require provenance for models and inference runtimes and apply hardware protections where feasible to restrict unauthorised GPU usage.
Quick checklist for a fast hardening sweep: remove unnecessary local inference runtimes; limit model cache write permissions; lock GPU device nodes to service accounts; require process sandboxing for any local model; enable prompt and output logging; and add IOA rules for model-driven persistence attempts. None of this is magic. Strong safety alignment in models raises the bar for attackers, and inconsistent model behaviour is an advantage for defenders while jailbreaks remain immature. Treat on-device LLMs as a new tooling layer that needs the same operational controls and sceptical scrutiny you apply to any privileged service.
Additional analysis of the original ArXiv paper
📋 Original Paper Title and Abstract
Living Off the LLM: How LLMs Will Change Adversary Tactics
🔍 ShortSpan Analysis of the Paper
Problem
The paper investigates living off the land LOTL style attacks in which threat actors abuse legitimate tools and processes already present on a system to avoid detection. It considers a plausible future in which on device large language models are used to automate attack chains and operate alongside trusted system utilities, enhancing stealth, persistence and evasion. The work situates this risk in the broader context of LOTL techniques that blend with normal system activity, and notes the security implications for critical infrastructure, governance and societal safety. It also foregrounds potential mitigations such as restricting LLM capabilities, strict sandboxing and isolation, least privilege for LLM workflows, monitoring and auditing prompts and tool usage, anomaly detection for AI driven actions, and secure supply chain and hardware protections. A broad societal security lens is taken to signal AI enabled crime and covert surveillance risks even if the primary focus remains technical threat modelling.
Approach
The authors synthesise existing LOTL research and provide a preliminary proof of concept for a local LLM based attack. They emphasise a scenario where an attacker already has access to a user profile and cannot rely on remote network based LLMs, instead exploiting whatever locally deployed LLM infrastructure exists on the victim system. The approach includes enumerating locally available models and resources such as GPUs, Python environments, Ollama, llama cpp and cached HuggingFace models, selecting a most capable model through a priority list (for the PoC gemma3 6b), and establishing a feedback loop in which prompts and function definitions are fed to the local LLM to generate code. The attack demonstrates a pipeline that can perform persistence and data manipulation entirely offline, and discusses how jailbreak style prompts can be used to overcome safety alignment. The article also surveys detection paradigms including command detection and indicators of attack IOAs, contrasting proactive IOAs with reactive IOCs, and outlines several defensive mechanisms tailored to LLMs such as prompt firewalls, output sanitisation, anomaly detection and restrictions on tool use and LLM enabled code generation.
Key Findings
- On device LLMs can act as autonomous agents enabling multi stage, fully automated attack campaigns that extend beyond single step code generation and can occur without external network calls.
- The PoC shows a locally hosted attack that first detects local LLM resources, selects a capable model, applies a jailbroken prompt to coerce code generation, and finally executes malicious actions including creation of a startup persistence service and covert deletion, all without contacting external APIs.
- Feasibility is strongly influenced by the safety alignment of the discovered model; robust refusal mechanisms can block many payloads, whereas weakly aligned or uncensored models lower the barrier to jailbreak and enable more potent actions.
- Model variability creates an operational challenge for attackers, who may need model specific jailbreaks and may face limited success against some families while succeeding against others; this uncertainty can be both a risk and a defensive advantage until jailbreak techniques mature.
- The threat space expands to supply chain and OSS ecosystems, where LLMs could be used to extract MITRE ATT&CK tactics from malicious packages or embed LOTL capable components in supply chain malware, increasing risk of broad, scalable abuse.
- Beyond direct attacks, LLMs lower barriers to complex social engineering such as autonomous voice phishing, and enable more scalable and sophisticated campaigns with reduced required expertise, potentially impacting critical infrastructure and organisational security.
- Defensive guidance emphasises defence in depth including prompt firewalls, output sanitisation to block tooling usage, anomaly detection, restricting tool access, usage controls for code generation, and crowdsourced rules to detect abuse patterns.
Limitations
The work presents a preliminary proof of concept conducted with a specific local model (gemma3 6b) in a controlled environment using Ollama, so generalisability to other locally hosted LLMs and deployment contexts remains to be established. The attack relies on the attacker identifying and exploiting existing local LLM resources and the model safety alignment at hand; results may vary across model families and versions. The PoC focuses on a single attack vector and does not claim broad real world prevalence, though it signals potential risk as local AI deployment grows. The authors acknowledge uncertainty about how practical such attacks would be in diverse enterprise environments and under different defensive postures.
Why It Matters
The paper highlights that LLMs residing on devices could significantly augment threat actors by enabling autonomous, multi stage campaigns with less human expertise, increasing stealth and persistence. It stresses that the emergence of robust on device LLM safety alignment acts as a security layer, whereas uncensored models present higher risk and may be more susceptible to simple prompts. The analysis points to practical mitigations including restricting LLM capabilities, implementing sandboxing and least privilege, auditing prompts and tool usage, applying anomaly detection to AI driven actions, and strengthening the security of supply chains and hardware. It also signals broader societal and governance implications as AI enabled crime and covert surveillance become more feasible, underscoring the need for proactive detection strategies such as prompt firewalls, output filtering, IOA based monitoring, tool usage restrictions and crowdsourced defence patterns. The work thus contributes to a forward looking risk assessment for defenders and industry, urging vigilance and layered controls as local AI tooling becomes more prevalent.