Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback

Key points

• Problem: In auto-research, judging whether AI-generated ideas actually work is hard. Most prior studies only had humans or LLMs rate idea quality, and AI-Scientist ran experiments on simple self-constructed datasets that limit comparison with existing methods in the same field. Prior idea-generation work also lacked a feedback mechanism that passes experimental results back into idea generation.
• Input and output: This is a system paper with an implementation and experiments. DOLPHIN takes a research topic (for example, 3D classification) and reference code, reads the titles and abstracts of related papers to propose ideas, modifies the code to follow an experimental plan and runs it, and feeds ideas that improved performance into the idea-generation prompt of the next loop.
• Method: An LLM extracts the task attributes of the topic (model inputs, outputs, and so on) and scores papers found through Semantic Scholar from 1 to 10 to keep only relevant ones. An idea is dropped as redundant when its maximum cosine similarity to the idea bank exceeds the 0.8 threshold and must pass an LLM novelty check, and when code fails, custom-code function names, lines, and code are pulled from the exception traceback to build a local code structure before debugging. Results are sorted into improvement, maintenance, and decline against the reference code, and ideas similar to earlier ones that failed to improve performance are filtered out in the next loop.
• Implementation: Idea generation and result analysis use gpt-4o-2024-08-06, code editing and debugging use deepseek-v2.5 served through ollama, and aider calls the LLM agents. The total number of retrieved papers is 50, each loop generates 20 ideas, and debugging runs at most 5 times. Per its README and file list, the GitHub repository (InternScience/Dolphin) holds the launch_dolphin scripts, dolphin_utils, examples, and docs, and the data for the three classification tasks is downloaded from a Google Drive link.
• Results: With two loops (40 ideas) per task, the best idea raised ModelNet40 OA (overall accuracy) from 91.0% for the authors' PointNet implementation to 93.9%, CIFAR-100 accuracy from 81.2% to 82.0%, and SST-2 accuracy from 91.0% to 92.5%, and the ModelNet40 result is comparable to the 93.8% of the human-designed GPSFormer. On the insult detection task of MLE-bench, which is built from Kaggle competitions, DOLPHIN raised code generated by AIDE (a code generation framework) from 81.0 to 84.7. In the per-loop analysis, the share of successfully executed ideas that improved performance rose from 2/7 in Loop 1 to 4/8 in Loop 3.
• Limits: The authors state that knowledge the LLM retained from training data may leak into idea generation, and that using only titles and abstracts can miss the technical details of papers. Current LLM coding ability is too weak for project-level code, so DOLPHIN cannot yet verify complex tasks. The 3D comparison excludes pre-training and voting, and the experiments span three classification tasks (point cloud, image, sentence) and three MLE-bench tasks.

Paper links

External research summaries. These are not HDATF publications or measured product results.