CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

Key points

• Problem: Existing LLM causal reasoning benchmarks generally turn causal graphs, datasets or narratives into question answering and classification tasks and usually provide the variables, evidence, graph or query up front, leaving the 'causal parrot' concern that models succeed through memorized causal facts or linguistic cues. The interactive Auto-Bench primarily asks whether an agent discovers a hidden DAG (directed acyclic graph), and the authors argue that evaluation should also measure whether the discovered mechanism transfers to a new instance.
• Input and output: This is a benchmark environment for interactive causal discovery by LLM agents. In each episode the agent receives prior measurement records generated by a randomly sampled hidden SCM, intervenes on one controllable property of a manipulator crystal at a time within a fixed budget and observes the results, and then predicts the resonance frequency of a separate reactor crystal governed by the same SCM. At every step it emits a DSL record with five fields, memory, thought, past data, hypothesis and the next planned experiment, and the hypothesis is parsed into a graph, a frequency equation and coefficients for scoring.
• Method: A DAG is sampled over the observable properties and frequency, root nodes take exogenous source values and the other variables are computed in topological order with linear or quadratic structural equations; the graph, equations and coefficients are shared by the records, the manipulator crystal and the reactor crystal, while controllable-property base values differ. Interventions are shift-style, replacing the variable's base term with v and keeping parent contributions, in place of hard do(X=v), and some graph families add a hidden disturbance H that is resampled after each intervention and added to a subset of variables. Scoring reports reactor frequency accuracy separately from graph precision, recall and F1, SHD (structural Hamming distance, the count of missing, extra and reversed edges), frequency-coefficient F1 and root-node identification.
• Implementation: The authors built a new SCM generator, intervention and reactor mechanics, DSL logging and scoring code on top of the 2D game-style DiscoveryWorld environment, and a DSL record that fails the schema check is re-prompted up to two times with the parser error. The main suite ran four models, GPT-5-mini, GPT-5.2-high, Qwen3.5-Thinking and Qwen3.5-Non-thinking, on graphs with 3 to 7 nodes, with up to 50 topologies per (graph size, model) cell and one run per task, at temperature 0.1 and a fixed budget of 2 observations and 4(k-1) interventions for a k-node graph. Observation-intervention scaling used GPT-5-mini and GPT-5.2-high on 4-node and 6-node graphs, and the other follow-ups primarily used GPT-5-mini.
• Results: With the 50 four-node topologies fixed, replacing the linear mechanism with a hard-quadratic one cut GPT-5-mini accuracy from 48% to 24% and frequency-coefficient F1 from 0.589 to 0.251, while root-node F1 rose from 0.559 to 0.829. For GPT-5.2-high on 6-node graphs, 20 observations with no intervention gave 92% accuracy and 0.471 all-edge F1, while 5 observations plus 20 interventions gave 80% and 0.799, and a Golden intervention chain that leaves a small set of candidate graphs explaining the data (Markov equivalence class) raised GPT-5-mini 4-node accuracy from 48% to 90% but lowered all-edge F1 from 0.793 to 0.728. Among the four models GPT-5.2-high had the best accuracy and lowest SHD at every graph size, yet it reached 64% accuracy and SHD 4.761 at 7 nodes, and adding one verification step before answering raised GPT-5-mini 4-node accuracy from 48% to 60%.
• Limits: The authors frame CausaLab as a controlled synthetic benchmark whose 3 to 7 node SCMs with mostly linear mechanisms and one hard-quadratic extension cover part of the causal structures, functional forms, latent variables and measurement processes found in real scientific domains. Experiments cover a finite set of LLM agents, prompts and interaction budgets, performance may differ with other models, tool interfaces, decoding policies or longer exploration budgets, and the main analyses focus on final predictions and final hypotheses. Interventions are shift-style in place of perfect hard-do operations, the main suite ran each task once, and the authors name overgeneralizing success on templated SCM tasks to real scientific, medical or policy settings as the main risk.

Paper links

External research summaries. These are not HDATF publications or measured product results.