I, too, can make a chart of perplexity drops
Today, OpenAI released a blog post showing the perplexity of their models over time. The drop is impressive, and the claim is that perplexity itself can be a universal measure of intelligence.
After seeing this chart, the thought that came to mind was: perplexity against what? Perplexity is a measure of how well token likelihoods predict the actual next token. Perplexity is meaningless without knowing what text the model is trying to predict.
The answer was buried in a footnote. The chart is based on models predicting a fixed corpus of code.
So the problem here is obvious. Could this chart just be a reflection of OpenAI models being increasingly trained on code?
Keeping the model the same but changing the training mix
I set about investigating this. My plan was to find an open source repo with a tiny model and a documented training set, then monkey with it. While getting started, my agent found an existing paper that already did this. (The agent was Astra 6.2, funny enough. It must have low perplexity on this kind of tomfoolery.)
The authors of the paper ran five training sets on a 160M model, varying only the proportion of the training data that was code. Here is what they found:
Spoiler: the “internal model” here is the same 160M model with 75% of the training corpus being code.
As expected, if you look at perplexity on non-code text, it increases dramatically:
Conclusions
Obviously, the model here with 75% code in the training corpus almost certainly is smarter at code. The point here isn’t that OpenAI’s chart measures nothing, only that their post should have a lot more qualifications [1].
[1] Funny enough, OpenAI does qualify in a single footnote about GPT-2 and GPT-3, saying “how well they predict code is a poor proxy for their general capabilities.” Why they didn’t expand this to the rest of the models is anyone’s guess.


