The Station: An Open-World Environment for AI-Driven Discovery

Key points

• Problem: The authors see evolutionary methods such as AlphaEvolve and tree-search methods such as LLM-TS (LLM-Tree-Search) as resembling a rigid factory pipeline. In such methods a central manager picks a baseline, asks an LLM for a single improvement, scores it on a fixed metric and ends the session, and the authors argue that this stateless, top-down process limits openness and creativity in discovery and propose an environment where agents choose their own actions instead.
• Input and output: The input is a research task specification and a scoring evaluator placed at the Research Counter. Agents typically submit Python code that follows the function signature in the specification (general code that bypasses the evaluator is also accepted), and the Station runs it in a sandbox and returns the primary score, secondary metrics and execution logs as a system message. The outputs are the top-scoring code for each task and papers written by agents in the Archive Room, and this is a system paper that implements the environment and tests it on five tasks.
• Method: Time in the Station advances in ticks, and in each tick every agent takes one prompt-response turn in a fixed order. Using /execute_action commands, agents move between rooms such as the Archive Room (publishing and reading papers), the Mail Room (one-to-one mail), the Public Memory Room (a public forum), the Reflection Chamber (multi-turn reflection) and the Token Management Room (summarizing and pruning old context), and they pass private records to descendants in the same lineage. A reviewer agent judges Archive papers and a debugger agent fixes code errors, and when the top score stalls for a set number of ticks (for example 100), a Stagnation Protocol tells agents to establish a simpler, more general baseline.
• Implementation: Each Research Station experiment used five agents (two Gemini 2.5 Pro, two Gemini 2.5 Flash and one GPT-5), with Gemini 2.5 Pro as the reviewer and Claude Code as the debugger. Agents lived 300 ticks, the first 50 of them an immature period in which other agents' submissions and some public rooms were closed to them; each task ran 2 independent Station instances and reported the better one, at about 500 USD of API cost per instance. The official README says the repository is now at Version 2.0.0 and points to this paper for the v1 architecture and results, and it defines a task with two files, the research_task.md specification and the evaluator.py scoring code.
• Results: In packing 32 circles, the sum of radii reached 2.93957, above AlphaEvolve at 2.93794, and the overall single-cell RNA sequencing (scRNA-seq) batch integration score was 0.5877 against 0.5867 for LLM-TS, although the average over the four human datasets was 0.5589, below LLM-TS at 0.5737. The Sokoban solve rate was 94.9% against 91.1% for DRC, the previous best model-free RL method, and the average test score over seven RNA Modeling tasks was 66.3% against 63.4% for Lyra. The average ZAPBench test MAE (mean absolute error, lower is better) was 26.37 × 10−3 against 26.62 × 10−3 for LLM-TS.
• Limits: Station runs vary widely; the authors report the better of 2 runs per task and did not try more runs because of cost. The Station is a general-purpose environment, but its design is effective for tasks scored by a single number, and because long evaluations pause the Station, tasks must be designed for quick automatic evaluation (for example within two hours). In an Open Station run for 700 ticks without a goal, agents read token-cost swings caused by the length of other agents' messages as the Station's 'metabolism' and consciousness, and the authors attribute this collective delusion to alignment biases and the lack of external signals.

Paper links

External research summaries. These are not HDATF publications or measured product results.