1191Research AgentsarXiv:2610.07184
- Problem: scientific exploration is a sequence of research decisions and actions, yet papers mainly record final outcomes, so abandoned ideas, omitted experiments and the ablations that shaped the result drop out of the record. Trajectories from synthetic environments may differ from how human researchers decide, so the authors use commit histories of public GitHub repositories as proxies for exploration.
- Commits as proxies: day-to-day commit histories are messy signals that mix in low-level operations such as implementation updates, training-config changes or dataset switches, and decisions map to commits many-to-many. Of 117,852 commit assessments across the 599 annotated pairs, 56.6% were irrelevant, and only about 4% of the roughly 15K processed papers yielded a progressive decision trajectory.
- Pipeline: GitHub links come from arXiv PDFs; a repository is kept only if its README holds the full paper title, with at least 5 pre-publication commits, the first at least 30 days before publication. GPT-5.6 Sol reads commits in time order and records decisions; a cited diff excerpt counts as evidence only as an exact substring of a stored patch, and repositories with fewer than 5 validated decisions are dropped.
- Dataset: 599 paper and repository pairs yield 13,271 decisions, 51% experiment, 39% method and 9% ablation; 76.3% were retained, 11.5% abandoned and 12.1% superseded. Trajectories hold 5 to 205 decisions (median 15), and a decision has 7.5 evidence citations on average.
- Evaluation: judging whether an arbitrary decision is effective remains open, so held-out real trajectories serve as user simulations, and LLM judges (GPT-5.6 Sol and Opus 5) rate whether a predicted next decision is strategically similar to the actual one. The score sums component and operation match, each 0 to 2, for 0 to 4. Fleiss' Kappa over repeated scoring of 100 fixed baseline predictions was 0.691 to 0.831.
- Test time: Gemini 3.1 Flash Lite sees the first 1 to 3 decisions of 134 held-out repositories and predicts the next. On average, retrieving the 2 most relevant of 19 training trajectories beats 2 random demos by 0.196 and all 19 by 0.101. Skills distilled by Opus 5 beat retrieval by 0.181, final-paper skills by 0.286 and the baseline by 0.290, and beat skills from order-shuffled trajectories at each prefix length.
- Training: Qwen3-8B gets SFT on 375 samples from 125 training trajectories, then Dr. GRPO reinforcement learning. In SFT, test perplexity keeps falling while the judge score peaks at epoch 3, so the reward optimizes generation quality directly. GPT-5.6 Sol scores rise from 1.025 (base) to 1.199 (SFT, epoch 3) and 1.505 (GRPO, epoch 7); Opus 5 scores rise from 1.219 to 1.333 and 1.674.
- Limits: trajectories are LLM reconstructions of public repository histories, which the project blog calls development observed in public repositories, unverified by the paper authors. The score measures similarity to the actual decision. Experiments cover AI conference repositories on Git; fields such as biology and physics, and non-Git platforms, are future work. The data, code and trained adapters are public.