Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Key points

• Problem: AI systems have made discoveries when given clear evaluation metrics, but whether they can solve open-ended tasks without such metrics is still an open question. In open-ended tasks, agent judgments may diverge from human researchers, and the streetlight effect appears, where agents pick easy questions and justify them as important.
• Input and output: This is an empirical study that extends Station, an existing multi-agent research environment, and evaluates it on five tasks. In three rediscovery tasks, agents receive only the research question and experimental setup (checkpoints, environments) of oral papers from ICLR 2025 and ICLR 2026, the oracle papers, with their results withheld and web access disabled. The output is a research report of about 8,000 words written by a Consolidation Agent, scored against the oracle findings split into about ten sub-discoveries, while two exploratory tasks without oracle papers are reviewed by humans.
• Method: In Station, 6 agents built on different models spend 300 ticks choosing their own research directions, running experiments, publishing internal papers and collaborating by mail. A Supervisor agent collects research proposals and keeps each agent on its proposed direction for at least 60 ticks, and Meta Reflection makes GPT-5.5 reassess each agent's research from an outside researcher's view every 50 ticks. At the end of a run, a Consolidation Agent built with Codex and GPT-5.5 combines the findings and evidence into a report.
• Implementation: Each run starts with 3 Gemini 3.1 Pro agents, 1 GPT-5.5 agent and 2 Claude Opus 4.8 agents, and Codex (GPT-5.5) writes and runs experiment code in a sandbox without web access. The full evaluation spans 50 research runs, over 3,400 hours of experiment time and 760 million output tokens; a GPT-5.5 agent scores the reports, and its votes agreed with a blinded expert on 64 of 66 judgments (97.0%). The official repository contains the station code, five task packages, run scripts and a web interface, with large datasets and checkpoints distributed separately.
• Results: Across the three rediscovery tasks, Station recovered 62.7% of sub-discoveries on average, above Codex Multiagent-v2 at 15.4% and AI Scientist-v2 at 14.4% to 20.6%, with 75.8% on emergent planning, 70.4% on low-rank structure and 41.9% on RNN temporal representations. Removing both Supervisor and Meta Reflection cut recovery on emergent planning from 75.8% to 57.9% and the average research project duration from 67.0 to 55.5 ticks. In an exploratory task, an intervention found by the agents, scaling the MLP outputs of the last two layers by 0.5, raised the correct answers of InternVL3.5-8B-Instruct from 28 to 35 on a 40-example set.
• Limits: An author of the emergent-planning oracle paper rated 24 agent proposals that did not overlap with the paper and gave 75% of them 1 or 2 out of 5, so agent research judgment still diverges from human judgment. The models have knowledge cutoffs from January 2025 to January 2026 and the earliest oracle paper appeared in April 2025, so prior exposure to its findings during training cannot be ruled out. The authors state that the compared systems also differ in model composition and workflow, so the gains cannot be attributed specifically to decentralization.

Paper links

External research summaries. These are not HDATF publications or measured product results.