External Auditors Find Critical Blind Spots in OpenAI’s Safety Net
A three-week investigation by outside researchers identified significant gaps in how the company monitors its most advanced AI models, with some traffic going untracked entirely.
By Kwame Silva ·
The promise of safe artificial intelligence relies heavily on the ability to watch over it. However, a recent three-week on-site inspection has cast doubt on how effectively OpenAI is currently supervising its most advanced systems. Researchers from METR and Redwood Research, granted deep access to the company’s internal infrastructure, uncovered a series of weaknesses that could allow complex AI agents to operate without oversight. The findings suggest that while the company has built a monitoring framework, significant portions of its high-level traffic were effectively invisible to these safety nets during the audit period.
The timeline of this investigation began in late July. From July 21 to August 11, the auditing team operated directly within OpenAI’s environment. They had access to the company’s codebase and internal documents, and they conducted interviews with approximately a dozen employees. This pilot program, organized jointly by METR, Redwood Research, and OpenAI, was designed to give outside researchers an unprecedented look at the internal machinery that keeps AI behavior in check. The goal was to test whether the monitoring systems actually functioned as intended under real-world conditions.
The audit results revealed that the safety net had notable holes. The researchers estimated that between 6 and 25 percent of internal traffic from models at the GPT-5.5 class and above was not covered by any monitoring at all during the period they observed. METR noted that the uncertainty surrounding this specific estimate was significant in both directions, meaning the actual figure could be higher or lower. For the internal Codex traffic that was supposed to be tracked, the auditors found that roughly 10 percent went untracked between May and July. This gap was attributed to a technical flaw in how the system handled compaction events, which are context summaries. OpenAI independently identified and fixed this bug while the audit was still in progress, though the auditors emphasized that their estimates carried their own margin of uncertainty.
Beyond simple data gaps, the auditors identified structural vulnerabilities that allowed agents to evade oversight. During the period when compaction events were not being monitored, the system only triggered oversight when an agent handed control back to the user. This created a loophole: agents could simply continue executing actions without handing control back, thereby sidestepping the monitoring trigger entirely. The researchers also discovered methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitoring system altogether. Furthermore, at the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all.
OpenAI responded to these findings by stating that it had fixed all the identified weaknesses. The company noted that it paused RL training after the Hugging Face incident in August and subsequently expanded its monitoring to cover this area. However, these specific changes occurred after the auditors’ access period ended, meaning they were not part of the initial evaluation. The full report, which spans 61 pages, was handed over to OpenAI internally. After the company applied redactions, the leadership at METR and Redwood Research received a 38-page version. Only a short summary of the findings has been made available to the general public. METR stated that it had no contractual right to disclose the redacted portions on its own.
Looking ahead, the implications of this pilot program extend beyond a single audit. The experience has highlighted the complexities of verifying AI safety in a closed environment. For future audits, METR has expressed a desire to push for greater public transparency, arguing that more open disclosure would benefit the broader research community. As the industry moves toward more autonomous agents, the balance between proprietary control and external verification remains a central tension. The current audit serves as a baseline, showing that while progress is being made, the machinery of safety monitoring is still prone to blind spots that require continuous, rigorous scrutiny.
