Dwarkesh Patel and Jerry Han published the results of a set of controlled pretraining experiments on September 8. Between 2019 and 2025, at a compute budget of 1e19 FLOPs, compute-efficiency gains from data improvements came to 12.0x, versus 3.7x from model improvements — the former is 3.24 times the latter.
How the experiment was run
The two researchers split model recipes and training corpora into two separate axes by year, then trained on every cross-combination. The model axis runs from GPT-2 in 2019 to OLMo-2 in 2025, with GPT-3- and NeoX-style recipes in between. The data axis runs from OpenWebText in 2019 to UltraFineWeb in 2025, with The Pile in between. OpenWebText holds roughly 9 billion tokens; modern corpora are several orders of magnitude larger.
They tested five compute budgets — 1e17, 3.16e17, 1e18, 3.16e18 and 1e19 FLOPs — computed on the nominal basis C = 6ND. Evaluation used OLMES, which aggregates 10 relatively simple benchmarks, mostly multiple-choice question answering. They chose it over cross-entropy loss for a plain reason: the datasets differ across runs, so loss values aren't directly comparable between them.
Fitting the results to a linear regression — a model that simply adds a data term and a model term — explained 88% of the variance.
The annualized numbers are smaller
Converted to an annual rate, the compute-efficiency gain from model-side improvements is 1.24x per year, from data-side improvements 1.51x per year, and 1.57x combined.
1.57x is below prior industry estimates. Anson Ho and colleagues put the average at 3x — nearly double this figure. The authors don't sidestep the gap, and attribute it to two factors: their experiments are too small in scale to surface many gains that only show up at larger scale, and the small-scale training setup doesn't capture inference-side optimizations either.
The two sets of numbers aren't directly interchangeable. One is a controlled, small-scale experiment with clean variables and limited room for extrapolation; the other is a macro estimate for the whole industry, broad in coverage but unable to separate out where the gains come from. The value of this work is in weighing "data" against "model" separately — the specific 3.24x figure is secondary to that.
A caveat the authors leave open
One line in the original piece is worth remembering more than the conclusion itself:
"if the gains are limited, then the main driver of pretraining progress will stall, because we're not generating more internet, and you can only curate a fixed set of data by so much."
In other words: if the gains are limited, the main driver of pretraining progress will stall, because we aren't generating more internet, and there's only so much you can extract by curating a fixed pool of data.
The authors list four limitations of their own. First, the experiment scale is too small to capture many gains. Second, the compute multiplier is fairly sensitive to which model recipe and which corpus are chosen — swap either and the results change. Third, they only studied cleaning methods applied to Common Crawl, and didn't cover three other paths: sourcing high-quality data from new sources, human-expert-generated data, and synthetic data generation. Fourth, swapping in a different benchmark suite "would look quite different."
The third point matters most. The incremental data sources that have come up repeatedly in public discussion over the past couple of years fall squarely into those three untested categories. This experiment answers how far six years of cleaning internet data has gotten pretraining — it doesn't answer how much runway is left beyond that.
Sources: Dwarkesh Podcast, OLMES benchmark documentation, CocoLoop; compute-efficiency multiples and annualized rates follow the original 1e19 FLOPs basis, and the industry 3x estimate is cited from research by Anson Ho and colleagues.