<- Back
Comments (195)
- jrfloThis trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day.Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.
- ACCount39Interesting that GPT-4.5 is the only discontinuity on the graph.The oversized monster LLM has performed worse than CoT-trained o1 and o3 on the benchmarks, so CoT + RL clearly provided better "real world performance" improvements - let alone "price/performance."But if pre-training perplexity on held-out data really does measure something like "base model IQ" or "base model g factor", then it does support the hypothesis that 4.5 still had more "intellectual potential" than the o-series.So, if you were wondering why do AI companies train 10T+ monsters now, and are taking aim at 100T too? It's because scale works.
- tensorI'm a bit skeptical about the claim that particular abilities "emerge" at given perplexities. For instance, there are smaller models that are specifically trained on math, and I imagine those models have much higher general perplexity on other tasks.On the other hand, assuming you have appropriate training data for a task, then yes obviously perplexity is going to measure performance on that task. Being able to predict the next task token is obviously going to be a near perfect correlation with accuracy on the task.
- x313I thought perplexity on simple text data has been saturated for a while (as in perplexity is uncorrelated to task success - ex. https://arxiv.org/html/2410.23771v2).I suspect what's happening is their test dataset is actually just something really hard, like math papers or something. In that case perplexity would just be a direct measure of math knowledge, which means this blog would just be saying that better math knowledge predicts better math skills.
- londons_exploreMarcus Hutter/Matt Mahoney are looking like pretty smart men right now:http://prize.hutter1.net/More and more evidence is piling behind their theory that data compression and intelligence are one and the same problem.
- synapsomorphyThe big part still being left out, to me, is the size of the model itself.Minimum descriptor length and Kolmogorov complexity are always size(compressed data + compressor). Right now we're mostly just looking at the first part, but it's trivial to smuggle in things in the second part that can make the first part appear as low as you like.
- armcatI've been working in NLP for many years prior to the release of ChatGPT, and perplexity was generally associated with "human readability" rather than intelligence. So this is somewhat fascinating.There was an ACL paper from Tokyo University a couple of years ago that showed, across all languages, lower perplexity does not necessarily correlate with better "human likeness" [1]. I wonder if that's also the case here (where intelligence is concerned), that we are biasing too much towards English.[1] "Lower Perplexity is Not Always Human-Like", https://aclanthology.org/2021.acl-long.405v1.pdf
- theptip> I further propose common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone (such as the development of a successor to CoT monitors).Very positive to see this proposed from OpenAI. I hope Anthropic and GDM follow suit.I suspect it’s not sufficient, but definitely necessary.The one concern I have is if we pin pretraining perplexity, then there will be more pressure on RL for progress.It seems that pre-training actually led to fairly human-aligned models; the Overton window is in some sense latent in the corpus.On the other hand, heavily RL’d models exhibit more reward seeking behavior, and seem to be decoupling from the friendly assistant persona in certain contexts. And swarm RL (as OpenAI is doing) produces agents with quite inhuman tendencies to self-sacrifice for other instances (see Hugging Face).So, I hope we also pair this with some measure of alignment or pacing for RL, too.
- LudwigNagasenaMinimizing perplexity may be enough to in some sense match human-level intelligence that, combined with a computer’s ability to do the grunt work, leads to super-human results. But actually going beyond human intelligence would almost by definition imply that perplexity on human texts should increase.If RL is merely a way to elicit latent capabilities acquired in pretraining, as the article claims, that could mean that currently there is a wall that separates human-level AI from superhuman-level AI. And no one has figured out how to break it yet.
- thrw045Maybe I misunderstand the technicalities behind compression and prediction but there is something wrong with them in terms of LLMs. You can't really compress knowledge all the time.You can for example talk to ChatGPT about religion - just tell it that you are on a hard journey and you hope God is on your side or something like this, and it will become a full on preacher of the gospels. In other words it hasn't really compressed the scientific knowledge with the religious knowledge, it contains both of them. If there was this objective type of compression and prediction, you would predict that the LLM would reach some kind of universal information state, but that doesn't seem to be the case?The other thing is, maybe there is a way to compress answers to questions that are objective. Like if I ask ChatGPT to extract the audio from a Youtube video and save it as a wav file. Relatively objective right? Well sort, but there's a gazillion ways you can do that. You can write a script in an endless amount of programming languages, you can segment the task into an almost innumerable set of sub steps (for example different scripts for different steps in the process), so what exactly does compression of that information mean when it can be compressed and predicted in so many different ways?I don't really have a total answer here, I'm just thinking about all of these things and wondering.
- pelican0The chart appears to say that "GPT-6.2 Astra" is the model that had solved the millennium prize problem (Navier-Stokes).I haven't seen this claim elsewhere, it's a big deal.Anyone from OpenAI here, or in the know, who can confirm?
- hn-aarvinI've been watching 3Blue1Brown's series on compression and intelligence^[1], which seems to discuss similar ideas.In particular, his mention of an unknown floor for loss seems directly connected to Claude Shannon's entropy rate regarding compression—having to probe human brains to estimate the entropy of English at 1 bit per character, just as OpenAI doesn't know the exact entropy rate of its held-out codebase.[1]: https://youtu.be/qbsb9DpdZ6s
- riemannzetaLog-perplexity is cross-entropy, and its reducible part is the KL divergence between model and corpus. I've argued that the same quantity prices the gap between any two agents, human or AI: https://www.symmetrybroken.com/maintaining-divergence/The floor Pachocki can't locate is fundamental. No model can predict the next word better than its underlying agreement with the author's model.
- mkarrmannI'm curious what exactly each "base-model" here is.We've generally believed that most model upgrades aren't new pre trainings, but instead are better post trainings of a common pre trained model. But this graph suggests every release had a better base model.Have the public narratives around how these models are trained been misleading? Does each model release usually always include an extended, if not completely fresh, model pretrain?
- armcatIt's absolutely insane when you look at the period between GPT 3.5 (March 2022) and GPT 5.2 (December 2025) - more than 3.5 years, and then just look what happened this year - the rapid improvements almost on a monthly basis, and log-perplexity just taking a dive (conversely, "intelligence" skyrocketing). I'm wondering if December 2025 will be known as the beginning of something?
- andaiThe divisions they drew in the log graph are of course arbitrary, but GPT-4 (the original 8k context version from early 2023) was able to unminify JS, as well as produce an autonomous coding agent. (This was via ChatGPT, so I copy-pasted the code into files, but it both designed and implemented it.)So "useful coder" arrived about three years earlier, by my count.
- MrCheezeI always had the impression that model capabilities were stagnating around GPT-5.0, and this finally broke around the release of Claude 4.5 later that year. Seems like their own graph corroborates that things were pretty stagnant then. I wonder what changed?
- GodelNumberingIf you take the argument at a face value and follow, this also implies that any advances are largely correlated with base model sizes (note the latest drop is from Astra, not sol 6x). And importantly, to drop the log perplexity incrementally, you have to increase the base model sizes continuously and likely disproportionately.In other words, there is only so far you can go with the model of a given size before needing to increase it for smaller incremental gains.
- deadlypointerKind of expected, they are increasing number of parameters, and the model can fit the data better. Does not mean anything about "emergent capabilities" in my reading.
- nico> Detachment from actionable evaluations. Perplexity does not directly correspond to a task we care aboutThe post makes a good case that at frontier scale this doesn't matter much, because perplexity gains turn into new capabilities. However, this has been an issue for me when training local modelsSome agents wanted to optimize for perplexity, and we did all kinds of experiments, but at least for what I wanted (a small conversational chat model that can run in my browser, ~8M params), that was the wrong way to go. I could get excellent ppl numbers, yet still almost no coherenceWhat worked best, in my case, was to get better data, specifically, hand-crafted examples for the tasks I wanted the model to be good at. I'm guessing that perplexity might track capability when the model is big enough to turn the gains into general skills, but at a few million params it mostly buys you better-sounding word statistics
- djeastmSo the jump in intelligence was like a kid that keeps getting better at reading and then one day they suddenly can do word problems. Now the worry is the student will soon be writing the textbooks and we won't be able to check the work?
- WASDxDo they say that they measure perplexity at the pretraining phase? I thought most of the advancements have come from RL during posttraining. Would an older model with modern posttraining have the same perplexity while being significantly better? Then I don't see it as a meaningful metric.
- matteorasoWouldn't reducing the entropy too much lead to lack of diversity of thought? An idea that I've had for a while is that LLMs are held back by having the same mind, leading to modal collapse. Ideally, an AI swarm would have different approaches to a problem so they don't all end up trying the same thing.
- KotlopouOh, I always wanted to know how the entropy of English was looking! The human baseline is both surprisingly unclear and surprisingly better than even today's frontier models (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...). Though this is an extrapolation from 6 years ago, so take it with a grain of salt.There's also the Hilberg model where the entropy rate keeps falling towards zero as you consider longer and longer texts, and at least here (https://www.youtube.com/watch?v=5eqRuVp65eY) data from LLMs showed no floor. It would be unintuitive if it turned out that you can compress sensible English text to, say, O(sqrt N) length!
- Imnimo>The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot)Am I right in understanding this to mean that the code on which we're computing perplexity was all written pre-3.5, and so it almost certainly entirely human-written? In other words, this isn't a snapshot of today's internal code base which contains a great deal of AI-authored code.
- yewenjieI am having a really hard time imagining what qualitative difference would gpt-5 and gpt-6 have in predicting the next token? What kind of dataset is it? How big is it?
- XenophileJKOThis feels fundamentally wrong. As we move up the intelligence arc, I would expect perplexity to increase in general and only decrease on the pre train.I would expect the perplexity on the original pre train to increase as reinforcement learning on verifiable domains teaches the model lessons that did not exist in the pre train. In addition alignment and other concerns, unless the are consistently enforced in pre train data quality will also move perplexity up.
- lejalvI would like to see perplexity vs. model size
- rdedevFrom gpt 5 haven't the actual harness played an important role? Pretty sure no model is a cyber security expert by just thinking hard enough. I feel like this article is doing a reverse causation hereFor example consider this: would it be possible to build a model with such a low perplexity score but it be pretty bad at cyber security tasks? What if we removed all cyber security related data and RLVR from its training?The article presents an observation but treats it as causation
- criemenHere I was under the assumption that since GPT-4ish we mostly improved models via RLing, and that pre-training stayed (roughly) constant. Mind blown :)
- amluto> The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot).Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.
- joe_the_userQuized ChatGPT about the article...I had trouble seeing it in the article (it's hidden in footnote 3) but it seems perplexity/log-loss is tracked using the model's ability to track an unreleased code base of OpenAI.The log-loss involved in code has declined more for code than prose in all of the models but has supposedly declined everywhere. [1]At the same time, the decline of the latest models on a common sense benchmark might be meaningful: https://news.ycombinator.com/item?id=50588916[1] https://openreview.net/pdf?id=SHMj84U5SH
- am17anA good and common metric. It will be interesting to see if other labs agree, but thank you for arriving at a metric and following up on the slowdown talk. I do hope others pay attention.
- AlexCoventryI wonder if someone could back out estimates of their model sizes, using this graph and scaling laws.
- qphe95I wouldn't say perplexity is great metric because GPT-4 and o3 have similar perplexity on this chart but o3 was a massive step improvement. Pachocki has an incentive to push perplexity as one of the creators of the original LLM paradigm but I would say we have kind of moved on from that.
- coolontheinternPerplexity is definitely a useful metric for pretraining, but doesn’t mean much for post-training. In fact, low perplexity in a domain during pretraining doesn’t always translate to better behavior in post-training. Moreover, given that most of the recent progress and most of the frontier compute is going into post-training, I think it isn’t a very useful metric for intelligence.
- londons_exploreIs this perplexity per-token or per character or per byte?Also, is that graph assuming no changes in tokenizer if the measure is per-token?
- ameliusDo we have a clear picture now when the ability to generalize is achieved?
- cuttothechasewhere is Terra?Is there a public link of all their models and where they fall along this or is this cherry picking points to fit the curve?
- hansvmThat's a nice story, but is there any evidence 2yo models with modern harnesses and the same compute expenditures wouldn't have produced the same results? This is written as some combination of a sales pitch and post-hoc rationalization without even bothering to control for the most basic confounding variables. Even if true, TFA isn't convincing, and not offering any more information than the conclusion it isn't helpful for any other purpose either.
- ericfloI think "common limits based on perplexity" is a radical, offensive concept. A single clever individual could run afoul - you should have the right to be clever!
- andaiWhen we approach 0, are they gonna stop sounding cringe?Actually, trick question I suppose -- the base models already don't sound cringe, it's post-training that adds that!(Well, some of them do, because a lot of companies are putting synthetic data and LLM outputs into the pretraining phase (as well as bot-infused post-2020 common crawl, I suppose), but the early base models were fully cringe-free.)
- ranyumeI find it convenient for labs that the proposed industry standard is a metric that only labs can calculate, using their own datasets that they won't share.Clearly, if this should become an industry standard, public regulators must be able to audit both the datasets and how perplexity is calculated.
- TheArcane> Publishing our internal perplexity measurements took some consideration; it gives competitors a clearer picture of where we stand,very noble of you, _Open_ AI
- andsoitisI’ve always thought the French AI company Perplexity chose a very poor name. It is perplexing why you’d pick a name with negative connotations.
- thehamkercatOpenAI hasn't released any competent model since opus 5.5 came out, i wonder if these labs are deciding to pass the "best model" card every quarter to each other for good public perception of all labs or something, because OpenAI can't be this incompetent
- fortran77Perplexity goes back to Jelinek's IBM speech recognition group in 1977. Same Jelinek who supposedly said every time he fired a linguist, performance went up. This post is that quote with a bigger electric bill.
- jkoal73in two sentence: „this metric we have been optimizing correlates with the model performance. Big things will happen”
- post-itClaude, make a graph where the x axis is time and the y axis is a line heading toward something impressive, and then write me an article about it.
- bluepeterLet's admit this post has some SEO goals too!
- Retro_DevSaying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is.
- a3w5.3 Codex was not a "useful coder", wat?
- scuppernongthis seems to confirm that the model code named "Bel" which solved Navier-Stokes is in fact 6.2 Astra.
- fallingbanannaThis is not the first time I've seen a chart of "Inteligence on a log scale" related to AI.And it again feels equally dishonest/bullshitty.You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.It's just marketing talk. In reality all three are separate skills and the real intelligence requirements are, if you ask me, not that far away from each other.To put this in an analogy, it feels like charting "athleticity on a log scale" and putting first running a marathon, then swimming across English channel, then benchpressing 200kg, each one presented as requiring an orther of magnitude being more athletic.
- hooloovoo_zooHmm every annotation but the millennium one could be applied to every point....and possibly that one too lol.
- BetelbuddyThis has a central argument around perplexity without ever properly defining it mathematically....Then talks about cross-entropy but fails to explain the relationship between entropy, log-perplexity, and perplexity itself.For an argument that relies so heavily on a quantitative metric, omitting such a fundamental explanation is poor.
- RAWGODWell, if you accidentally leak the evaluation set into the training set, you can reach perplexity 0.Or the training set might just grow to be so big, defining a truly independent evaluation set is almost impossible.
- anonundefined
- LearyLooks like their latest internal model performs somewhat similarly to model 4 of anthropic.
- f3408fhInteresting that they work on weekends. Is it just because they're all basically millionaires living the dream so they don't care? Every place I've worked at has discouraged working late and weekends. Could also just be a scheduled post.
- eulgroDidn't they hit a data wall roughly two years ago and had to rely on artifical datasets for pretraining i.e. LLM-generated?I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?
- bigmadshoe> Given that the main way new surprising capabilities emerge is by progress on compressionIs it just me or does the article completely lack proof of this premise? I see a plot of perplexity dropping over time, with the implication that performance has risen as a result. This is just a correlation. What else did they change about post-training during this timeframe? This doesn't feel like a real experiment or real science.
- zurferThe chart pretty much explains why I was freaking out about AI at GPT-4, because the trend of improvement has been so steep. And then followed a strong slow down, but now in 2026 things have really accelerated again. My hunch is really that coding is a freaking important input for AI and we might have actually entered some form of recursive self improvement. I know some people hate the term but I feel it's somewhat fine as coding is saturated (models can get better at it) and it has many levers for model training and inference (from algorithms to data).Anyway, I think it's good idea. Would love to see all models on this chart, but honestly we might not be able to trust it going forward as it becomes a target.
- anonundefined
- orbital-decayPlain perplexity as a 2026 metric? It has to be marketing.
- FergusArgyllInteresting that he's attributing so much to the base model. I thought the majority of the gains are due to post training
- rfgplkWhat's the perplexity of the average person?
- pishpashI don't understand why anyone needs to exponentiate existing, well known concepts like entropy and cross-entropy to invent something called "perplexity". Somebody has Shannon envy syndrome.
- tonyhart7what ??? there is no release model ????I am sorry OpenAI, but I think we are seeing OpenAI fall from grace hereAntrophic basically got out before AI bubble burst and Dems block most of the data center build out
- blueblisters[dead]
- oxcartctl[dead]
- natsu529[dead]
- twotwoone[dead]
- GIFtheoryThe post makes perplexity seem overly mysterious… First, “log perplexity” is just cross entropy. Why not call it that?Also, when perplexity is this low, it arguably makes more sense to report perplexity. A perplexity of 2 basically means that the model can guess a set of size 2 that contains the true next token.
- hypferThis is just an ad. Why is it top #1 of the front page?Just some guys on a payroll claiming that an arbitrary set of milestones is a curve towards a greater magic goal that means that some valuation is actually true.Why would you read this
- jwpapiI honestly mean that I don’t know, but an alternative way of reading this is:Anthropic is getting uncatchable by us and we merely keep ahead against chinese models, so let’s all slow down.I think the reason Anthropic is so far ahead could be that they get way more user conversations, as everybody else distills, the rest is one gen behind...
- yipinwongThey are touching an elephant's legs, and saying Elephant is like a log.
- AIShill101Gah man do I hate AI ethics and stuff. I really hoped the lead scientist at openAI wouldn't be so screwy but they're all part of that cult aren't they?I really hope we go full speed ahead on AI development.