Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent

Published
Source
arXiv
Paper number
1470
Field
Agents
arXiv ID
2611.17939

Key points

  • Seven identical runs of the agent on 18 questions passed between 105 and 123 of 149 rubric criteria, and the bill varied by up to 17%.
  • Classifying 20 failed trajectories under a pre-registered taxonomy assigned 13 to the agent's instructions, 3 to its tools, 4 to the model, and none to its data.
  • Four blind readers named the same fault as the analyzer in 75 of 80 cases, supporting the diagnosis's trustworthiness.
  • Under a cost objective, the final configuration reached 0.34× the baseline cost on the held-out questions at accuracy within noise.
  • The accuracy objective gained 14 points on the development questions but only 1.5 on the held-out ones, with a confidence interval straddling zero.
  • As a limitation, this is a case study of one agent and nine test questions, not a general result.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)