HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery

Key points

• Problem: Existing AI-scientist-style systems mostly aim at full automation with minimal human involvement, largely in ML research. Empirical economics needs questions that the variables in an actual dataset can answer, careful identification strategies (study designs that isolate causal effects), and human judgment of economic significance. The authors point out that ideation without dataset information often yields infeasible or hallucinated hypotheses.
• Input and output: This is a system paper with an implementation and experiments. The input is one of the supported local datasets (CHNS, the China Health and Nutrition Survey, a longitudinal survey; CMGPD-Liaoning, the China Multi-Generational Panel Dataset of historical demographic records; UK Biobank, a large-scale biomedical dataset with socioeconomic variables), public data APIs such as World Bank, FRED, and OpenAlex, and a research-domain constraint set by the researcher, such as labor economics. The output is a draft paper compiled from Markdown to PDF with Pandoc (a document converter) and LaTeX, together with regression tables, figures, result summaries, and replication files, and a human steps in at two decision gates: research question selection and publication approval.
• Method: A central orchestrator keeps a shared state object (RunState) and hands work to seven agents in the order data audit, data profiling, questioning, data collection, analysis, writing, self-critique, and review. Question generation is conditioned on dataset diagnostics such as the variable inventory, missingness patterns, distributions, and correlations, and passes through a question quality loop in which feasibility is screened and a human selects. After a draft is written, the Reviewer Agent scores it on a 1 to 10 scale and requests extra analyses such as robustness checks, driving a research revision loop of re-analysis and revision.
• Implementation: HLER is written in Python, all LLM reasoning runs on Claude Sonnet 4.6 through the Anthropic API, and econometric estimation uses the Python statistics libraries statsmodels and linearmodels for OLS (ordinary least squares), fixed-effects panel, difference-in-differences, and event-study models. The official repository maxwell2732/hler-working-papers holds numbered working papers produced with HLER (Markdown and PDF), an R_script folder, and an image folder, and its README states that the repository contains working papers only.
• Results: Over 14 runs on three datasets, dataset-aware question generation produced 69 feasible questions out of 79 (87%), against 34 out of 82 (41%) when the model got only a dataset name and domain description. 12 of the 14 runs (86%) finished end to end with no human intervention beyond the two decision gates, and for these 12 runs the mean Reviewer Agent score rose from 4.8 for the first draft to 5.9 after the first revision and 6.3 for the final draft. A run typically took about 20 to 25 minutes, with an average API cost of $0.8 to $1.5.
• Limits: The authors state that dataset-aware generation reduces infeasible questions but that the theoretical meaning and novelty of hypotheses still need separate checking, and that econometrics is limited to OLS, fixed effects, difference-in-differences, and event studies, leaving instrumental variables, regression discontinuity, and structural models for future work. Reviewer Agent scores come from the same LLM that wrote the manuscripts, so the evaluation may be circular, and the evaluation covers 14 runs on three datasets. Because many hypotheses can be tested quickly, multiple-testing and selective-reporting risks grow; the authors say this lies outside what HLER solves, so researchers must apply corrections and report the full set of questions.

Paper links

External research summaries. These are not HDATF publications or measured product results.