Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research

Key points

• Problem: Existing Deep Research and Wide Search systems typically operate within 10 to 20 retrieval iterations and about 100 web pages, and each mainly concentrates on either depth or width. How well LLMs handle questions that need long-horizon planning, massive evidence gathering and synthesis across heterogeneous sources remains largely unexplored. The authors judge existing evaluations ill-suited to such questions, citing alignment errors in LLM-as-a-Judge report comparison, shallow fact-recall metrics, costly human evaluation and neglect of uncertainty expression.
• Input and output: This is a benchmark paper. A system under test receives one expert-written question and returns a long cited research report, and the benchmark scores that report against a prebuilt Research Graph (fact and insight nodes linked by evidence) and a QA (question-answer) exam for that question. There are 300 questions across 10 domains, and the benchmark statistics give 97 reasoning steps on average (142 at most) and 601 retrieved web pages on average (1,267 at most) per question.
• Method: GPT-4o and a Gemini deep research agent drafted candidate tasks, and Ph.D. candidates and senior practitioners removed hallucinations and kept only tasks that are solvable through public retrieval, more complex than a simple keyword search, and unambiguously stated. The ground truth combines a Research Graph that humans and LLMs built from Planner, Researcher and Summarizer agent outputs by linking URL-anchored fact nodes to higher insight nodes, an expert-checked canonical report, and a QA exam. The score is a weighted average of depth-weighted node recall, the share of conclusions with a complete evidence path to facts multiplied by graph coverage, a QA exam answered only from the report, and stance calibration on contested topics, while single-source dependence is diagnosed separately.
• Implementation: 12 systems were evaluated: five deep research systems (Gemini, Sonar, Tongyi, o3 and o4-mini Deep Research), two search-integrated agents (Kimi-k2, Grok-4-1-fast), and five baselines that plug DeepSeek-r1, MiniMax-m2.1, Claude-3.5-sonnet, Qwen-2.5-72B and Llama-3.3-70B into a common LangGraph agent with Tavily search. The authors built a web tool for report viewing, graph projection, exam-item editing and final audit, and the abstract gives a leaderboard address.
• Results: Among the 12 evaluated systems, the top one, Gemini Deep Research, reached only 28.62 Overall, followed by Sonar Deep Research at 27.04 and Kimi-k2 at 26.16, while the LangGraph baselines clustered between 16 and 23. Gemini's Logical Consistency of 22.92 lagged its Coverage of 33.15, and the authors note that logical consistency appears to be a universal performance bottleneck. When 1 to 3 key facts or insights were removed from, or 1 to 3 relevant facts added to, 120 reports, the graph metric moved in the right direction 57.4% to 79.6% of the time, against 14.8% to 22.2% for an LLM judge.
• Limits: The authors note that such super-complex questions may be rare in standard applications and frame the benchmark as a ceiling evaluation. The sensitivity study used 120 reports from 12 models and 10 tasks, and in the evaluator-swap test the graph Coverage score fluctuated more (4.42%) than the LLM judge (3.88%). The authors also flag the risk that agents synthesizing massive evidence compound hallucinations and invite over-reliance on authoritative-sounding reports, along with high compute cost.

Paper links

External research summaries. These are not HDATF publications or measured product results.