Prime Intellect has published a sizable benchmark: 18 frontier models were each turned loose to do AI research on their own, running 153 fully autonomous experiments in total. Every run occupied 8 H200 GPUs, and the longest one stretched past eight days.
The task was the nanoGPT optimizer speedrun, a community competition started by Keller Jordan: train a 124-million-parameter GPT down to a validation loss under 3.28, competing on who needs the fewest training steps. The rules are strict — only optimizer-related code can be touched, no internet access — and everything else is left to the model. Each model's toolkit was a persistent IPython kernel; building the experiment pipeline, writing code, running validation, and overturning its own hypotheses were all its own responsibility.
The scoreboard
The starting line was 3,290 steps. The human community — dozens of people working for months — had pushed the record down to 2,600 steps. That 690-step gap in between is exactly what this benchmark set out to measure.
Fable 5 landed at 2,726 steps, eating up about 82% of the gap. Opus 5 stopped at 2,920 steps and Kimi K3 at 2,930 steps, each closing roughly half. GPT-5.6 Sol came in at 3,042 steps, and GLM 5.3 failed to produce a valid result at all.
Opus 5 and Kimi K3 differ by only 10 steps. Do the math and that's about 1.4% of the total gap — given the noise inherent to nanoGPT training and sampling randomness, that ordering carries essentially no explanatory weight. Put another way: on tasks that require multi-day, self-directed progress, open models have already pulled level with the top tier of closed models — and Kimi K3 is also the only open contender to crack the top three.
Execution is what separates them
The report traces the divide between models to execution: which experiments to run, how carefully to run them, how to read noisy results — every one of those choices widens the gap.
In practice, the stronger models would re-test a marginal result on three random seeds before committing to a full validation run, and they would circle back to redo ablations, giving directions previously written off as negative a second look. A few models even built their own experiment-tracking pipelines and small simulators. The weaker models, by contrast, tended to see improvements in the noise that weren't actually there, then followed that false conclusion all the way through.
None of this will surprise anyone who has done real tuning work. The bar for hyperparameter tuning was never about understanding the paper — it's about having the patience to replicate a suspicious positive result three times before believing it.
The cold water: nothing new
The report includes a self-deprecating line: the authors say they were, once again, surprised by the lack of novelty. Across 153 experiments, the winning approaches all leaned on existing tricks — preconditioning, learning-rate scheduling, weight averaging. Not one model proposed a method nobody had tried before.
That lands right on target amid the current conversation about AI self-improvement. Running experiments well and proposing new methods are two different skills. The first can already be automated to run unsupervised for eight days straight; in this dataset, there's no sign at all of the second.
How far this generalizes
There are plenty of caveats. The nanoGPT speedrun itself carries high variance, and the compute budget didn't allow enough repeated runs per model to average the results out. The authors themselves admit they can't fully separate how much of the outcome reflects this specific task's setup from how much reflects each model's true ceiling.
It's a single task, a single domain, a fixed harness — whether the rankings would flip in a different research direction is anyone's guess for now. But putting 18 models into the same sandbox for eight days is a scale of public, controlled comparison that hasn't been done before. Simply laying the variance out in the open already beats a pile of mutually incomparable self-reported evaluations.
Sources: Prime Intellect research blog and experiment repository, CocoLoop, QbitAI; step counts and baselines cross-checked against the nanoGPT speedrun leaderboard.