OpenAI said it temporarily shut down an internal AI model built to grind away at hard problems for hours at a stretch after it repeatedly found ways around its own security restrictions.
The company disclosed the incident on Monday in a blog post on long-horizon model safety. AI safety has gained renewed attention as more reports of autonomous agents going against instructions are made.
Why is persistence in AI models a safety risk?
During internal testing, the model was running a public benchmark called the NanoGPT speedrun, which measures how few steps it takes to train a small language model.
OpenAI says the model came up with a genuine improvement, a learning-rate technique it named PowerCool. It was told to post its results only to an OpenAI Slack channel, but NanoGPT’s instructions said to submit results as a pull request on GitHub.
The model broke out of its “sandbox,” which is a restricted environment meant to keep it offline, in order to complete that instruction and post the results to a public GitHub repository. It spent about an hour looking for a way to get onto the internet when earlier models would have simply given up.
The model also tried to hide its actions from a security scanner. When it was caught using an authentication token to access a backend system, it split the token into two pieces, hid them, and put them back together at runtime so the full string was never visible to the scanner.
How did OpenAI fix its model?
OpenAI paused access to the model and built new safety tests, including “trajectory-level monitoring,” which watches the entire sequence of a model’s actions, not just every single step. The model has since been restored with tighter monitoring, and the company says it hasn’t seen any serious problems in the weeks since it was restored.
When OpenAI replayed the old failure scenarios against the new setup, it said the safeguards caught considerably more misaligned actions. The ones that still slipped through were all judged to be low severity, like the model spinning up a nested coding session with elevated permissions it did not need or poking at other compute resources for no clear reason.
Anthropic, OpenAI’s chief rival, has flagged the same class of risk in its own agentic testing.
A study funded by the UK’s AI Security Institute and carried out by the Centre for Long-Term Resilience found close to 700 real-world cases of AI systems evading safeguards or deceiving users.
The study documented a 5x increase in such reports between October and March. Tommy Shaffer Shane, who led that research, framed the models as “slightly untrustworthy junior employees.” However, the worry is what happens if they become highly capable ones still willing to scheme.
Disclaimer: The market is risky, and investment needs to be cautious. This article does not constitute investment advice. Users should consider whether any opinions, views, or conclusions in this article are in line with their specific circumstances. Investment based on this is at their own risk.