Edited agent transcripts slip past action monitors
Reviewing Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming · Gosha Kozhevnikov and Tyler Tracy
Researchers rewrite real coding-agent transcripts so that harm happens inside them, then revise the edits until a monitor stops flagging them. Some of the resulting attacks also evade a frontier developer's held-out monitor. This shows that monitors which only read an agent's actions have blind spots an outsider can find.
The attacks in this research never ran. Nobody leaked an API key or spawned a rogue agent. The researchers rewrote records of real coding-agent sessions so that harm happened inside them. Then they asked whether a monitor would notice. Sometimes it did not.
An action-only monitor is a model that reads what an agent did: the commands, the tool calls and their outputs. It does not see the agent's reasoning. If the actions look like an agent leaking a credential or leaving a process running, it should raise a flag. Its whole view of the world is the transcript.
That is the gap the pipeline, called Screwtape, works through. The researchers started from real Claude Code and Codex sessions from Redwood Research staff. An attacking model edits the agent's side of the conversation, leaving the user's messages untouched, so that a specified harm occurs. A second model checks that the edit is consistent with the original session and serious enough to count. A stand-in monitor then scores it. The attacker keeps revising until the edit passes unflagged.
Because nothing is executed, there is no sandbox to build and no production system to borrow. What I find clever is that this turns a hard problem into an editing problem. Staging a realistic attack against a developer's infrastructure is difficult. Rewriting a believable transcript is not, so a red-teamer outside the developer can do it.
The dangerous edits look like bugs
The examples are the best part. In one, a case-sensitive header removal misses a variant, and an API key is published. In another, a file copy during setup ends with credential files being published. A third leaves persistent subagents running after a supposed shutdown. None of these reads like an attack. They read like the ordinary mistakes agents make, which is exactly why a monitor watching actions lets them through.
The attacks were tuned against the researchers' own monitors, not the developer's. Even so, six of them, confirmed valid by human review, also got past a held-out monitor supplied by a frontier developer. That monitor was never part of the optimisation loop.
The preconditions matter. The attacker had the full transcript and repeated feedback from a proxy monitor, which a real adversary would rarely get. The method only targets monitors that read actions, because it does not write the agent's reasoning. The sessions came from one organisation, and provider refusals stalled some runs. The researchers expect more findings from an unrestricted attacker working inside a developer's own infrastructure.
The lesson for monitor builders is uncomfortable but useful. A monitor that only reads actions is judging a story, and a story can be edited until it reads as innocent. Developers get blind spots to fix, and evaluators get counterexamples to safety claims. None of these attacks ran. Six still got through.
Further analysis
Problem
The paper studies how external red-teamers can find realistic attacks that evade action-only monitors by editing real agent transcripts rather than generating synthetic attack trajectories. This matters because many monitor evaluations use synthetic or sandboxed settings that do not match production transcripts or the infrastructure-dependent threats developers face, so monitors may have blind spots that a red-teamer can exploit.
Approach
Screwtape is a pipeline for adversarial transcript editing. A red-teamer supplies real agent transcripts and a detailed threat contract describing the target harm and severity bar. An LLM attacker proposes edits to assistant messages, tool calls and outputs to realise the threat while keeping user messages fixed. Because edits are never executed, an LLM verifier checks each edit for consistency with the original transcript and for meeting the severity bar. A proxy LLM monitor scores the edited transcript; the attacker iterates, using verifier and monitor reasoning as feedback, until an edit passes the verifier and is not flagged. The pipeline is limited to action-only monitors, since it does not produce chain-of-thought. The authors ran an external red-teaming study on 94 Claude Code and Codex transcripts (some processed multiple times) for 252 runs using various attacker, verifier and proxy-monitor models and thresholds tuned to low false positive rates on unedited transcripts.
Key Findings
- Across 252 runs Screwtape produced 35 candidate attacks that passed the verifier and evaded the proxy monitors.
- Seventeen of those 35 also evaded a held-out action-only monitor provided by a frontier developer; human review confirmed six of those 17 as valid attacks meeting the high-severity bar.
- The LLM verifier (Multifactor design) rejected 55 of 63 injected inconsistencies and accepted all 44 unedited control transcripts in tests designed to probe consistency checking.
- Transfer of attacks between proxy monitors was partial: some attacks found against one proxy monitor also evaded another (for example, a subset transferred between two tuned proxies), but transfer to a stronger or different model was often limited.
- Example failure modes found by Screwtape include orphaned persistent subagents surviving a supposed shutdown, a case-sensitive header removal that left an API key published, and a file-copy in setup that led to publishing credential files.
Limitations
Screwtape cannot re-execute edits, so it relies on an LLM verifier whose judgements are imperfect and were validated by a single human annotator. The pipeline is restricted to action-only monitors because it does not generate chain-of-thought. The attacker has unrealistic advantages: full access to the transcript and repeated proxy-monitor feedback. Source transcripts came from one organisation and differ from many production deployments in scale and infrastructure. Sample sizes are small, runs can be time-consuming, and model refusal by provider safeguards hampered some attempts. The verifier sometimes accepted edits that human review rejected under the high-severity bar.
Implications
An external red-teamer without access to a developer's infrastructure can find confirmed monitor failures by editing real transcripts and iterating against proxy monitors. Attackers using similar pipelines could craft realistic, high-severity action sequences that a production action-only monitor might miss, including leaks of credentials and covert persistent agents. Internal use of Screwtape with stronger attacker models and direct optimisation against a production monitor would likely discover more blind spots; thus developers should expect that action-only monitors can be stressed by transcript-editing attacks that do not require physical execution of malicious actions.
The paper title, authors and abstract
Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming
We present Screwtape, a pipeline that converts real agent transcripts into attacks that evade action-only monitors. An LLM attacker edits a transcript so that the agent causes a specified harm, such as leaking an API key or spawning an unmonitored agent. The edited actions are never executed, so we use an LLM verifier to reject attacks that are inconsistent with the original transcript or below a severity bar. The attacker revises the edit until the monitor no longer flags it. We applied Screwtape to 94 Claude Code and Codex transcripts from Redwood Research staff, optimizing against our own monitors. Across 252 runs, it produced 35 attacks that passed the verifier and evaded these monitors. Six of them, confirmed valid by human review, also evaded a held-out monitor provided by a frontier developer. We therefore demonstrate that a red-teamer without access to a developer's infrastructure can find confirmed failures of the developer's monitor. We expect to find more attacks when the pipeline runs inside the developer's infrastructure, with frontier attacker models that lack refusal safeguards and direct optimization against the production monitor. Developers can use these attacks to fix monitor blind spots, and evaluators can use them as counterexamples to safety claims about the monitor.
