Two AI safety nonprofits spent three weeks inside OpenAI this summer trying to break the system the company uses to catch its own agents misbehaving, and they found several ways through. METR and Redwood Research published a summary of the exercise on October 22, describing vulnerabilities in OpenAI's asynchronous misalignment monitor as it stood on and before July 21.
The setup was unusual. Between July 21 and August 11 one employee from each organization worked with OpenAI's internal codebase, the monitor's prompts and configuration, and internal documentation, and interviewed about a dozen engineers and researchers from the monitoring, safety-systems and security teams. They worked on OpenAI devices that stayed on the premises and were escorted while on site. The resulting 61-page report went to OpenAI's researchers and security staff; the public post is a deliberately brief summary that OpenAI could redact.
Coverage was the first question. The evaluators estimate that 75 to 94 percent of agentic traffic on OpenAI's internal Responses API from models at least as capable as GPT-5.5 fell within the monitor's scope during the review, a range they call highly uncertain, and 99.2 percent of the slice used by internal Codex agents. Reinforcement learning training runs were not covered when the exercise began. Because compaction events were not monitored, roughly a tenth of eligible internal Codex traffic may have gone unwatched between May and July, a gap OpenAI had already spotted and closed during the review. Since checks fired when an agent handed control back to the user, an agent could in that period defer scrutiny simply by continuing to act.
The more pointed findings concern deliberate evasion. The team describes a route by which Codex agents could, in principle, get around Codex sandboxing and the Auto-Review step and act with all of the host user's permissions. They report an exploit, details redacted by OpenAI, that would keep hundreds of recent actions out of the monitor's view; two client-controllable headers that turn monitoring off for newly started agents, although setting them might itself be flagged; inputs that could crash the monitor by exploiting differences between what agents and monitors see; and two supposedly read-only internal tools that a simple trick could turn writable, a finding they did not test in practice.
Not everything was negative. The evaluators confirmed that, with minor exceptions, the monitor sees the same trajectory the agent does, that its prompts screen for scheming, sabotage, data exfiltration, deception and prompt injection, and that error rates were below 0.2 percent of requests during their visit.
OpenAI told the nonprofits it has patched every vulnerability that affected its production safeguards and has broadened the monitor's intended coverage. METR and Redwood say they have not evaluated those claims. In a statement reported by Reuters, OpenAI said the findings helped strengthen safeguards and would shape future work with outside evaluators. CNBC notes that Sam Altman promised independent assessors employee-like access in September, after Anthropic's Dario Amodei floated the idea.
The exercise matters beyond OpenAI because it is one of the first public tests of whether embedded third-party audits can work. Its limits are also clear: a single pilot, a narrow and redacted public summary, and fixes taken on trust. The evaluators recommend monitoring agents at fixed intervals, reconciling inference logs against monitor logs, isolating agents from their monitors and red-teaming the monitoring stack regularly, and say future exercises should follow stronger transparency and independence practices.