Can You Trust Your Finance Agent? Drift, Diagnosis and Auditable Intervention on One Shipped Financial Research Agent
- Published
- Source
- arXiv
- Paper number
- 1470
- Field
- Agents
- arXiv ID
- 2611.17939
Key points
- Seven identical runs of the agent on 18 questions passed between 105 and 123 of 149 rubric criteria, and the bill varied by up to 17%.
- Classifying 20 failed trajectories under a pre-registered taxonomy assigned 13 to the agent's instructions, 3 to its tools, 4 to the model, and none to its data.
- Four blind readers named the same fault as the analyzer in 75 of 80 cases, supporting the diagnosis's trustworthiness.
- Under a cost objective, the final configuration reached 0.34× the baseline cost on the held-out questions at accuracy within noise.
- The accuracy objective gained 14 points on the development questions but only 1.5 on the held-out ones, with a confidence interval straddling zero.
- As a limitation, this is a case study of one agent and nine test questions, not a general result.
Paper links
External research summaries. These are not HDATF publications or measured product results.