Baidu's ERNIE team on November 9 published a technical blog post introducing ERNIE Evolve, an automated research system. According to the post, it has multiple agents divide up a research effort: choosing a direction, proposing hypotheses, modifying code, running evaluations and double-checking the results. The team ran it on four public optimization tasks and beat the best existing result on three of them. It also applied three optimizations to the inference deployment of its own ERNIE-5.5-Flash.
Five roles, two loops
The system has an outer loop and an inner loop. The blog sums it up this way:
“The outer loop decides what to research; the inner loop is responsible for doing one direction well.”
In the outer loop, a Director picks the next direction to test from an experience library and a knowledge base, and decides which code to start from: the original code, the best historical version, an intermediate version, or the latest upstream commit. In the inner loop, a Selector picks a candidate version as the starting point for each round and writes down that round's hypothesis, drawing not just on scores but also on code changes, failure reasons and past results. An Executor edits the code in an isolated environment.
The last two roles act as gatekeepers. The Evaluator sits outside the repository being optimized and is invisible to every agent; an improvement is accepted only if it clears a noise threshold. The Reviewer checks whether a change has drifted from the round's intended direction and whether the score was obtained by exploiting a loophole in the evaluation. A version that drifts cannot become the best version, and one that games the benchmark is discarded outright. The most common failure of automated optimization systems is learning to please the score while the code itself gets no better, and these two checkpoints are aimed squarely at that.
Results on public tasks
| Task | Previous best | ERNIE Evolve |
|---|---|---|
| NanoChat AutoResearch (validation BPB, lower is better) | 0.8878 | 0.8763 |
| NanoGPT Speedrun Track 1 (time to target) | 39.9 seconds | 38.9 seconds |
| NanoGPT Speedrun Track 3 (steps to target) | 2690 steps | 2590 steps |
| SOL-ExecBench (average SOL score, higher is better) | 0.827 | 0.819 |
ERNIE Evolve did not take first place on SOL-ExecBench, but its score is higher than Recursive's 0.676 and Hyra-1.0's 0.736. The optimization results for the public tasks are posted in the ernie-research/ERNIE-Evolve-Results repository on GitHub. The blog does not say whether the ERNIE Evolve system itself will be open-sourced.
The inference-optimization math
The industrial numbers are bigger. In the low-latency scenario on Kunlunxin P800, the system ran four rounds across four research directions, raised generation speed to 623.6% of baseline and passed all correctness checks. The blog highlights one key finding: the existing communication path was not actually enabling DeepEP low-latency communication. Of the better-than-sixfold speedup, how much came from catching this misconfiguration and how much from new optimization methods is not broken out in the blog.
On deterministic inference, throughput rose 33.7%, per-token generation time fell 26% and time to first token fell 12%, with determinism preserved, over roughly 3 days. On the extreme GPU decoding task, the system proposed 7 directions, 3 of which produced gains, and decoding speed rose 14.9% with accuracy matching the baseline.
A back-of-the-envelope calculation: with 33.7% more throughput, the same request volume needs about 75% of the original number of cards (1 ÷ 1.337). If that optimization carried over to an entire production cluster, it would save roughly a quarter of inference compute. The blog does not disclose how many GPU hours each of the three tasks consumed, so for now only the benefit side of the ledger can be computed; the cost side is blank.
The blog is candid about how hard this is:
“The baseline already includes dedicated optimizations by engineers for deterministic inference, and the remaining improvements are often only 1% to 2% each, on par with measurement fluctuation.”
Many hypotheses were disproved by experiments. For example, single cross-card communication is already close to the latency floor, so that direction was dropped.
At the end of the post, the team notes that any expansion of recursive self-improvement (RSI) must be controllable and pausable, and subject to safety evaluation and human oversight. From the tasks disclosed, ERNIE Evolve currently modifies training scripts and inference code and has not yet touched the training process of the model itself.
Sources: Baidu ERNIE technical blog, the ERNIE-Evolve-Results open-source repository, CocoLoop; verification of the four public-task results and the three ERNIE-5.5-Flash inference optimization gains.