PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaboration

Key points

• Problem: LLM-based multi-agent systems (MAS) execute experiments well within predefined workflows, but they often generate hypotheses that lack clear direction and find it hard to establish links between hypotheses and the evidence that supports or refutes them, which makes exploration inefficient. Many systems are also tailored to specific tasks, often relying on prompt engineering heavy with domain knowledge, so moving them to a new scientific domain often needs substantial modification.
• Input and output: This is a method paper that presents a plug-and-play module for existing MAS together with its theory. Inside the MAS, a Hypothesis Agent proposes one testable hypothesis per iteration grounded in a selected principle, with a major and a minor premise, and an Experiment Agent calls an experimental tool (here a surrogate model) to obtain a quantitative outcome such as a property value. PiFlow takes the accumulated principle-outcome pairs as input and gives the Planner agent the highest-potential principle with one chosen instruction: refine, validate or explore alternatives.
• Method: Principle selection is framed as a Min-Max optimization that combines exploitation, which minimizes cumulative regret (the summed gap between the optimum and the obtained value), with exploration, which maximizes the mutual information gained from a hypothesis outcome, and the authors give a proof in the appendix that with finite entropy and a bounded evaluation function the bound on cumulative regret grows sublinearly, at the order of the square root of the number of steps T. In practice, an exploitation score from each principle's observed outcomes normalized to 0 to 1 and an exploration score from the cosine distance between sentence embeddings of a principle and already tested principles are mixed with λ to pick the best principle. With the default thresholds, if that principle's exploitation score is above 0.7 the instruction is refine, above 0.4 validate, and otherwise explore, and with fewer than 3 records the system first initializes one principle to explore.
• Implementation: In the main comparison, all agents use QwenMax as the base LLM with external search tools blocked (the model ablations swap the LLM of the Hypothesis Agent and Planner for models such as GPT-4.1-mini and Qwen3-32B), and locally deployed surrogate models serve as the primary evaluation tool in place of real experiments. The tasks are Nanohelix Optimization (NHO), where a LightGBM model (coefficient of determination r2 0.98), trained with 5-fold cross-validation on 80% of 6300 records of DFT (density functional theory) simulation data and tested on the remaining 20%, predicts the chirality measure g-factor, Molecular Bio-activity Optimization (MBO), where a GNN (graph neural network, test-set r2 0.91), trained on 80% and tested on 20% of 50,000 molecules randomly sampled from the bio-activity compound database ChEMBL35, predicts pChEMBL (a negative log of molar concentration expressing activity), and Superconductor Optimization (SPO), where an MLP (multi-layer perceptron, test-set r2 0.91), trained on 21,056 and tested on 5,265 of 26,321 superconductor records, predicts critical temperature, and each searches for the candidate that maximizes the target within 24 budget steps. Per the README, the official repository holds a conda environment file, inference.py for a nanohelix demo task, per-task configs, a tools folder for registering one's own experimental tools and a Colab tutorial, and running it needs an OpenAI-compatible API key.
• Results: Averaged over the three tasks, PiFlow reached 49.86% AUC and 65.41% SQ, above The AI Scientist v2 (38.01%), the baseline with the highest average AUC, and AI Researcher (58.16%), the baseline with the highest average SQ. Per task, AI Researcher's SQ on MBO (95.66%) exceeded PiFlow's (84.55%), and The AI Scientist v2's AUC on SPO (28.43%) exceeded PiFlow's (21.51%). On NHO, the Vanilla agent took 2416 seconds to reach 50% SQ while PiFlow reached 52% SQ in 427 seconds, 5.6 times faster, and in the cost analysis, which ran each task for 24 iterations with QwenMax, token use against Vanilla fell by 26.7% on NHO and 13.7% on SPO but rose by 7.7% on MBO.
• Limits: Locally deployed surrogate models served as the primary evaluation tool in place of real laboratory validation, and the authors state that applying the approach to a new domain first requires building a reliable simulator or predictive model. From varying λ on NHO with QwenMax, the authors conclude that exploration efficiency is sensitive to the choice of λ and that the optimal value is task-dependent, and because exploration scores compare all pairs of past principles, the per-iteration cost grows with the square of the number of records times the embedding dimension, which the authors admit can become a bottleneck for very long processes (over 24 NHO iterations the PiFlow part took 9.4% of the total PiFlow-MAS runtime). The standard deviation of PiFlow's SQ on MBO was a large 29.63, Bayesian optimization with a manually defined search space beat PiFlow in the 24-iteration comparison on NHO AUC (68.86%), NHO SQ (78.76%) and SPO AUC (31.61%), and most ablations were run on the single NHO task.

Paper links

External research summaries. These are not HDATF publications or measured product results.