OpenAI Chief Scientist Jakub Pachocki has published a post titled "Perplexity and Intelligence" on November 22, 2026. In it he argues that perplexity, or equivalently cross-entropy loss, should become a metric that model developers compare with each other and use to inform shared safety standards. As a first step, he says OpenAI is publishing internal measurements of its pretraining progress, a decision he describes as a trade-off because it gives competitors a clearer picture of where OpenAI stands.

Pachocki frames modern AI as the result of training a compressor, or predictor, on a very large corpus, followed by techniques such as prompting, fine-tuning or reinforcement learning that draw out the capabilities of that model. Perplexity measures how well a model predicts the next token on average, and OpenAI has tracked it on a held-out dataset drawn from its internal codebase. The figures he reports are a log-perplexity of about 0.99 for GPT-3.5 and about 0.91 for GPT-4, with the most recent models now below 0.8.

He ties those thresholds to what models could do. According to the post, GPT-2 could write coherent paragraphs, GPT-3 could learn in context, GPT-3.5 held fluent conversation, GPT-4 used chain-of-thought reasoning productively, and models below 0.8 have shown new abilities in cybersecurity and mathematics. Pachocki says it was not clear in advance that these abilities would appear at those precise values. He adds that after OpenAI moved to a model at about 0.76, it began producing notable results on open mathematics problems, and that a similar jump could reach pretraining research as soon as 2027.

On that basis he expects further transformative capabilities in the 0.68 to 0.7 range, which he says may not be tested for today. He also expects current forms of alignment auditing to degrade and awareness of being evaluated to grow. He calls for frontier developers to agree on shared norms for evaluating such models, for red-teaming safety cases before first use, and for transparency standards. He further proposes common limits, based on perplexity and other measurements, that developers would commit not to cross until their ability to understand and monitor AI behavior reaches a concrete milestone, such as a successor to chain-of-thought monitors.

The post lists the weaknesses of the metric itself. Labs use different evaluation datasets and data mixtures, so numbers are hard to compare. Perplexity does not map directly onto tasks people care about, does not capture gains made after pretraining such as chain-of-thought reasoning, and has an unknown floor. Pachocki still argues that it has held up over time and can give the industry a common way to talk about progress.

All figures and proposals in the post are OpenAI's own and come from a single first-party essay. The post is a call for industry norms, not an announced rule or agreement, and it does not say whether other labs have agreed to publish comparable numbers or to accept shared limits.

References