AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists

Key points

• Problem: Building AI co-scientists requires high-quality data-driven discovery tasks for training and evaluation, but existing data-driven discovery datasets are hand-built, contain only a few hundred tasks, and serve evaluation only. Annotating one task takes trained graduate students at least 2.5 to 3 hours, and scientific analysis tasks need complete file-level code that runs on real scientific datasets, which makes them harder to mine automatically than software tasks that can often be extracted from pull requests. The authors also note that agents with promising results mostly rely on proprietary LLMs, which hinders their adoption in fields that need transparency and data privacy, such as social science and medicine.
• Input and output: This is a dataset paper that releases an automatic collection pipeline together with the training dataset it produced. The pipeline takes only a few seed keywords per discipline from a human and turns research code in GitHub repositories into pairs of a task instruction and a standalone executable code solution, plus a folder of the needed data and dependency files. The resulting AutoSDT-5K has 5,404 tasks from 1,325 repositories and 756 Python packages across four disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology and Cognitive Neuroscience.
• Method: AutoSDT-Search expands the seed keywords with an LLM, searches GitHub and PapersWithCode, keeps only research code repositories in the target discipline by having an LLM read each README, and removes repositories that overlap with the evaluation benchmarks. AutoSDT-Select drops files over 1,000 lines and directories such as config and tests by rule, then has an LLM pick files that meet three criteria (scientific workflow functionality, dataset input, scientific output) and extract the paths of their dependencies. AutoSDT-Adapt uses Claude-3.7-Sonnet to make the code run standalone, self-debugs it in a conda environment for at most three rounds and filters out failures, and then has an LLM back-translate the program into a task instruction.
• Implementation: GPT-4o (2024-11-20) handled most stages and Claude-3.7-Sonnet handled code adaptation; processing 2,993 repositories into 5.4K tasks cost 2,955 USD in API fees, or 0.55 USD per task on average. AutoSDT-Coder is Qwen2.5-Coder-Instruct 7B, 14B, and 32B with full-parameter fine-tuning in LlamaFactory, trained on 4 H100 GPUs for 7B and 14B and on 8 for 32B. A footnote states that the code and data are public on the project page (osu-nlp-group.github.io/AutoSDT).
• Results: Nine experts reviewed 256 tasks and judged that 93% of instructions describe meaningful real research tasks and 92.2% of programs are correct solutions. With zero-shot direct prompting and the average of 3 sampled responses, AutoSDT-Coder-32B reached a 7.8% success rate on ScienceAgentBench, double the 3.9% of its base model and on the level of GPT-4o (2024-05-13) at 7.5%, while the 7B model fell from 3.3% to 2.3%. On DiscoveryBench, the hypothesis matching score (HMS) of the 32B model rose from 6.9 to 8.1, while GPT-4o (2024-05-13) scored 10.4 under the same re-implementation.
• Limits: The authors state that AutoSDT produces task instructions and code solutions and leaves automatic generation of per-task evaluation scripts to future work, which limits uses such as reinforcement learning, and that outcome-based verification is hard to automate because research code comes without unit tests to use. Because of resource constraints such as API cost and access to experts, only 2,993 repositories were processed, and the scope is Python code in four disciplines. Generating long chain-of-thought (CoT) rationales at scale was difficult, so training reasoning models was left for future work, and a large gap remains on ScienceAgentBench against reasoning models such as Claude-3.7-Sonnet at 18.6% and o1-preview at 23.9%.

Paper links

External research summaries. These are not HDATF publications or measured product results.