RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents
- Published
- Source
- arXiv
- Paper number
- 1401
- Field
- Research
- arXiv ID
- 2610.36878
Key points
- The authors built RubricAtlas from 3,079 Chinese financial research reports, yielding 99 research types, 921 questions, and 8,160 checkpoints.
- An individual report covers a median of 54% of its type’s questions, so relying on a single report narrows the evaluation.
- Two judge pipelines achieved checkpoint agreement of κ = 0.71 and agent-ranking agreement of Kendall τ = 0.87.
- Deleting the blocks answering one question lowered its score from 0.64 to 0.21, while adding 40% irrelevant text produced no score increase.
- Evaluating only questions supported by the task’s source report raised completeness scores by 10.5 points and changed rankings, so the choice of standard shapes the outcome.
Paper links
External research summaries. These are not HDATF publications or measured product results.