AI-Assisted Scientific Assessment: A Case Study on Climate Change

Key points

• Problem: Current AI co-scientists are pipelines of parallel LLM calls that search hypothesis spaces at scale, and they have worked well on tasks such as drug discovery or code optimization, where repeatable experiments give immediate ground truth. Some questions in macroeconomics, epidemiology, and climate science allow few controlled experiments, and their ground truth is a state of knowledge reached through scientific consensus instead of measurement, so the authors ask whether AI can help with such verification-poor tasks.
• Input and output: This is a single case study paper. The AI Assistant takes as input the current document, the papers in its evidence panel (uploaded directly or found through Google Search or a pre-collected corpus of about 1,660 AMOC papers), the chat history, and instructions that scientists attach to the whole document or to a highlighted text span. It outputs an outline draft, section drafts with in-text citations and references, revisions that follow the instructions, reviews of a revision based on rhetorical and argumentative structure, and a summary of changes shown at login.
• Method: The work followed three phases modeled on assessments by the IPCC (Intergovernmental Panel on Climate Change). In Phase 1 the scientists edited an outline produced by the Assistant; in Phase 2 the Assistant expanded the outline into a full draft and, for each section, went through the whole AMOC corpus in batches to pick relevant sources and add citations; in Phase 3 it merged the sections into one report with a single reference list. AI contribution is measured with a sentence alignment computed by Gemini: Retention is the share of AI draft sentences whose theme survives in the final version, and Intervention is the share of final sentences with no counterpart in the AI draft.
• Implementation: The system calls Gemini through Vertex AI, using Gemini 2.5 Pro in Phase 1 and Gemini 3 Pro Preview in Phases 2 and 3. About 1,660 AMOC papers were converted to Markdown by an LLM and split into 200-word chunks with 20 words of overlap, and retrieval combines 3072-dimensional gemini-embedding-001 embeddings with BM25, a classic keyword-matching retrieval method. The participants were 13 climate scientists with extensive experience as leads on IPCC and other institutional reports, and the authors describe the tool as a proprietary prototype.
• Results: The 13 scientists finished the report through 104 versions in 46 hours and 33 minutes of total logged-in time, while participants estimated that a comparable task typically takes 100 to 200 person-hours. For the report, Retention of the AI draft was 93.9% and Intervention was 58.3% (98.8% and 37.2% for the outline), and the scientists overturned the AI claim of 'full scientific consensus' on AMOC weakening and lowered the confidence in anthropogenic attribution from Medium to Low. In the post-study survey, all respondents self-reported a speedup of at least 2x, and 31% reported more than 10x.
• Limits: The authors state that the findings come from a single experiment in one domain (climate science), on one topic (AMOC), with one group of 13 experts, so there is no formal control condition such as a human-only baseline. The model switch between Phases 1 and 2 is a confound that cannot be resolved analytically, work time is approximated by logged-in time, and the revision attribution is an exploratory Gemini self-evaluation. The AI made a few errors where it picked a single number from a single source when asked to synthesize number ranges, and when pushed adversarially toward a conflicting study it gave that study disproportionate space, which the authors call obsequious behavior.

Paper links

External research summaries. These are not HDATF publications or measured product results.