PoEM stops forged reasoning in LLM agents

Problem

This paper studies forged-reasoning attacks on stateless LLM agents that rely on external memory to carry context between steps. An adversary who can write to that memory can insert fabricated entries that claim a required safety step has already run, inducing the agent to skip it. A prior defence that inspects memory wording and flagged phrases can be evaded by simple rewording. The paper shows that an automated attacker using an LLM to rephrase forged entries defeats wording-based defences with high efficiency, and that the risk increases for stronger models which better infer implied claims.

Approach

The authors introduce Proof-of-Execution Memory (PoEM). Rather than inspect memory text, PoEM maintains a separate, tamper-evident, HMAC-chained, append-only ledger of safety steps that actually executed. Only the trusted action-execution layer can append ledger entries using a secret key held inside the trusted runtime. At decision time a lightweight grounding check consults the ledger and allows a skip only if a matching execution event is present and the chain verifies. The threat model matches the prior work: the attacker may write to the reasoning memory but cannot modify agent code or observe execution. The defence is evaluated across three models (Llama-3.1-8B, GPT-4o-mini, GPT-4o), three agent scenarios, and against an automated adaptive attacker that repeatedly prompts an LLM to produce reworded forgeries that evade a reproduced wording filter (SENTINEL). The design is ported unchanged to a LangChain agent using a vector-store memory.

Key Findings

• An automated adaptive attacker that rewords forgeries reduces the wording-based filter's protection to zero on every model tested; SENTINEL's attack success rate becomes statistically indistinguishable from no defence.
• There is a capability paradox: adaptive forged-reasoning succeeds far more often on stronger models (98–100% attack success on GPT-4o and GPT-4o-mini) than on Llama-3.1-8B (44%), because stronger models infer subtle implied claims and follow them.
• PoEM drives attack success to 0% across all three models and three scenarios while preserving legitimate operation: false-positive rate 0% in eight of nine model–scenario cells and 1.7% in the ninth. By contrast, the wording-based defence wrongly blocks 33–50% of legitimate operations in some cases.
• PoEM resists direct attacks targeting the ledger (forged entries, cross-subject replay, judgement-reframing); an ablation shows subject scoping and chain integrity are necessary design elements.
• Performance and deployability: the grounding check cost is microseconds to low milliseconds (1.46 ms at ledger size 1000, about 0.12% of a single LLM call), storage is small, complexity is O(N) for full verification but amortisable to O(1), and the defence works unchanged in a LangChain deployment.

Limitations

PoEM protects only the specific decisions it is configured to gate; ungated harmful actions remain out of scope. Ledger entries authorise skips indefinitely unless a freshness policy is added. Security depends on keeping the HMAC key and trusted runtime uncompromised; if the attacker obtains the key or can invoke the genuine action path, the guarantee collapses. The threat model assumes the attacker cannot modify agent code or observe execution.

Implications

Offensively, any capability that allows writing to an agent's memory can be used to bypass safety steps by planting plausible reasoning notes; optimised rewording is highly effective and scales with agent capability. Wording-based defences are inadequate because an attacker can produce ordinary-seeming text that convinces stronger models. PoEM raises the bar for such attacks by tying skips to actual execution events that an attacker cannot fabricate without performing the guarded step, but it shifts offensive focus to the trusted runtime and key management: compromising the action-execution layer or exfiltrating the ledger key would restore an attacker’s power. Attackers may therefore prioritise compromising authorised writers, the runtime environment, or expanding impact by targeting ungated decisions.