Baidu has unveiled ERNIE Evolve, an automated research system capable of proposing, editing, and testing code improvements for AI models. The system has outperformed existing benchmarks and successfully optimized inference speeds for the company's ERNIE-5.5-Flash model.

Key Developments and Technical Context

ERNIE Evolve operates through two nested loops, where a director selects research directions and parallel lanes evolve code trees.

The system employs a hidden evaluator and an independent reviewer to guard against reward hacking and evaluation loopholes.

In production tests, ERNIE Evolve increased generation speed on Kunlunxin P800 chips to 623.6% of the baseline after 4 research rounds.

The system beat the best public results on 3 of 4 public metrics, including NanoChat and NanoGPT Speedrun benchmarks.

Strategic Analysis and Industry Implications

The use of hidden evaluators and independent reviewers highlights Baidu's efforts to mitigate reward hacking, a common challenge in automated research systems.

The results achieved on ERNIE-5.5-Flash demonstrate the potential of applying automation to real-world engineering tasks rather than limiting it to academic benchmarks.

Outlook and Key Factors to Watch

Baidu has not disclosed the specific LLMs powering the agents within the ERNIE Evolve system.

The system remains closed-source, with only results and research logs currently available on GitHub.