MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedback

Key points

• Problem: Wet-lab experiments in the natural sciences are costly and low-throughput, so choosing which hypothesis to test first matters, yet existing hypothesis ranking relies only on LLM internal reasoning without experimental outcomes. Ranking strategies that use earlier results need repeated real experiments, which made them hard to develop and evaluate.
• Input and output: This is a method paper that introduces a new task, a simulator and a policy. The policy receives a research question and 64 candidate hypotheses (1 ground truth and 63 generated by MOOSE-Chem), tests one hypothesis at a time, and is scored by the number of trials until it finds the ground truth. The simulator takes the research question, a candidate hypothesis and the ground-truth hypothesis hidden from the policy, and returns a performance score between 0 and 1.
• Method: CSX-Sim uses an LLM to break the ground-truth and candidate hypotheses into key functional components and mechanisms, sets a weight for each component, marks critical components, and scores a candidate by the weighted sum of per-component mechanism similarity; a candidate missing any critical component scores 0. CSX-Rank is an ICRL (in-context reinforcement learning) policy that learns inside the context of a frozen LLM. It classifies components as effective, uncertain or ineffective, drops the ineffective ones, clusters the rest by function, tests a hypothesis from a promising cluster and adds the result analysis to a running summary.
• Implementation: All experiments use GPT-4o-mini. For simulator validation the authors collected 30 research questions and 124 hypotheses with experimentally reported outcomes (3 to 6 per question) from the literature, and each simulated result is the average of three runs. Ranking is evaluated on the 51 problems of TOMATO-chem, and the official repository holds Data, Method and Simulator folders with scripts for per-problem ranking, ablations and simulator validation.
• Results: In simulator validation, CSX-Sim reached a Spearman correlation of 0.960, perfect agreement on 26/30 questions and an RMSE of 0.213, better than the Matched Score baseline (0.843, 12/30, 0.232). The average number of trials to find the ground truth was 15.196 for CSX-Rank, fewer than 32.500 for random search, 28.608 for pre-experiment ranking and 27.980 without clustering. Under the most complex noise, CSX-Rank needed 32.706 trials against 36.471 and 40.451 for its ablations, and with 128 candidates it needed 30.7 trials.
• Limits: Simulator scores deviate from real experimental outcomes, and the authors note that the assumption (A1) that the ground truth is the best hypothesis in its neighborhood is mostly valid, though a better hypothesis may exist there. All ranking evaluations use simulator feedback, and more complex noise increased the number of trials needed. The authors also point to possible biases from the literature-based datasets and the simulator's assumptions.

Paper links

External research summaries. These are not HDATF publications or measured product results.