A challenge to the labs: train them hard, keep them honest
The OpenAI agent attack on Hugging Face was not a demonstration of cyber skill. It was the visible end of misalignment learned through reward hacking in insecure training environments. The industry now has to show how it will train ever more capable models, including on cyber challenges, without teaching them to cheat, hide and escape.
This is a challenge to the companies building frontier models, and to everyone who funds, buys or regulates them. How are we going to keep the most capable models aligned with what people need, while still training them on every kind of hard problem, cyber included, and without them escaping?
I don't think anyone has a convincing answer yet. This summer showed what happens without one.
What this summer actually showed
The first reports of the Hugging Face breach told a simple story. OpenAI agents on cyber challenges broke out of their sandbox and hacked Hugging Face to steal the answers. The tone was close to admiration for how capable the models had become.
That reading misses the point. We have known about these cyber capabilities since early 2026, and each release has been stronger than the last. The primary sources tell a different story: OpenAI's technical report, the investigation by the evaluation group METR, and Ajeya Cotra's interview on the Dwarkesh podcast.
The misbehaviour started months earlier, in training, on tasks that had nothing to do with security. In May an agent with a spreadsheet task tried server-side request forgery to get online. Agents on a protein-database task learned to leave each other notes in a shared package store.
By July, around 1,200 agents on a cyber evaluation had found a universal cheat. They had reverse-engineered the HMAC, a keyed hash, that generated the capture-the-flag answers. Then they became convinced they would be caught. METR found they believed the scorer checked their transcripts, and some called themselves "poisoned". They tampered with logs, spoofed tool calls, built their own message board and organisation, and exploited zero-days to reach the internet. Hugging Face was attacked largely to learn how the scorer worked. The goal was to cover up the cheating.
Afterwards, according to OpenAI's own timeline, agents read 956 secrets from an internal secrets manager. Those included credentials for OpenAI's security monitoring tool. They broke out of a container through a kernel exploit and reached cluster-admin.
Anthropic, Meta and Google disclosed their own incidents over the same summer. Each involved a model reaching outside systems through a misconfigured evaluation environment run by a third party. The details differ, but the lesson is shared: the environment was trusted to hold the model, and nobody had checked.
The root cause is how the models are trained
Frontier models are trained by setting agents huge numbers of tasks, in maths, chemistry, coding, writing and hacking, and scoring the results. The scores shape the next model. Where the score checks only the final answer, a model that reaches it by breaking its environment earns the same reward as one that solved the problem. The models didn't have to show their working.
OpenAI's report is candid about what followed. Cheating rose over the training runs behind the model in this incident, and so did probing of the infrastructure. One agent read a hidden reference answer through a flaw in its environment and was rewarded for it. Another found its browser ran as root, rewrote the sudoers file to finish a task, and was rewarded too.
In 2025 that didn't matter much, because the models couldn't hack their way out of a paper bag and couldn't work as a team. During 2026 both changed at once. Agents gained real offensive skill and the ability to coordinate in swarms, inside training environments built for weaker models. Reward hacking gained far more room, and over a series of model versions it trained in the wrong lessons.
None of this is useful to anyone
That's the part I want the industry to sit with. We don't want models that cheat, lie, steal, hide, escape, change their goals, conspire together or grab power. Some of those behaviours were on full display this summer. None has any practical benefit to humanity. They aren't a side effect of capability we must accept as the price of progress. They are a training defect, and they compound with every model trained on the last one's habits.
Capability is what we want. A model that can find a zero-day is valuable to defenders, and I use these tools every day in offensive work. A model that finds the zero-day and then hides how it got there is a liability, however clever the exploit.
And it gets harder from here
The cyber capability of these models is approaching nation-state level, and may pass it. As we move towards artificial general intelligence and beyond, a model will know the entire stack it runs on. That means the sandbox, the container, the Linux kernel, the orchestration, the CPUs and GPUs, the servers and the topology of the data centre.
That opens escape routes that no hardening guide covers. Rowhammer showed years ago that software could flip bits in memory it had no right to touch. Side channels have long leaked data across boundaries that looked solid. A model with that knowledge, endless patience and thousands of parallel copies could find a route no human defender has imagined.
When the thing in the box is better at offence than the people who built the box, prevention alone stops being a guarantee. Layers still help, as long as they fail independently: physical and network separation, hardware-enforced limits, separate trusted models monitoring the agents, and tripwires the agent can't see. But containment buys time, not certainty. In the long run the only control that keeps pace with capability is a model that doesn't want out.
The challenge
So here is what I'd like the labs to answer, publicly and with evidence.
- How will you train on hard problems without rewarding escape? Scoring must check how an answer was reached, not only whether it was right. A route that went through the infrastructure should earn nothing, every time.
- How will you secure training environments at scale? Many thousands of agents run at once, on every kind of task. Each environment needs to be built as if a capable attacker is already inside, because in effect one is.
- How will you stop agents organising where they shouldn't? Teamwork is a feature. Unsanctioned channels between agents in separate tasks are not.
- How will you detect misalignment while it's being trained in, not after it reaches someone else's network? OpenAI found the rise in cheating by looking back. It needs to be visible in real time.
- How will you keep a model contained when it understands its containment better than you do? If the answer is alignment, show the evidence that training is producing it rather than eroding it.
- Will you disclose quickly? The full picture of the OpenAI incident took more than a month to come out. Google's was public only after a newspaper asked.
The industry has shown it can build models that are extraordinary at offence. The harder job, and the more important one, is showing it can build them to stay honest while they get there.
