AutoRSE: Evaluating LLM Agents as Recommendation Engineers for Self-Evolving Recommender Systems

Key points

The authors built AutoRSE, an evaluation framework spanning six representative industrial-grade recommendation tasks, where in each task the agent receives a faithfully reproduced state-of-the-art algorithm, one H200 GPU, and a 12-hour optimization budget.
Across 42 evaluated settings of three frontier LLMs and three harnesses in seven combinations, agents outperformed the baselines in 35 cases, achieving a mean relative improvement of 16.1% across all tasks and configurations.
Trajectory analysis of 4425 segmented experiments showed engineering optimizations accounted for 91.7%, dominated by Model Ensemble at 59.0% and Parameter Tuning at 26.2%, while algorithmic innovations were only 8.3%.
Final submitted algorithms achieved a modest average novelty score of 2.42 out of 4.0, and higher novelty correlated strongly with larger gains at Pearson r = 0.79.
Injecting ideas from frontier research and human experts beat self-generated ideas by 13.35% in average improvement and 0.58 points in novelty, confirming that idea generation is the bottleneck.
Agents achieved an average gain of 29.2% on generative recommendation tasks compared to 3.1% on CTR prediction, so less saturated baselines left more room for agent gains.

External research summaries. These are not HDATF publications or measured product results.