Cache Engineering for Agents
- Published
- Source
- arXiv
- Paper number
- 1372
- Field
- Reasoning / Efficiency
- arXiv ID
- 2610.28577
Key points
- The authors measured that KV cache reads take up 82.4% of the total API cost on long trajectories.
- Cache engineering uses drop to exclude selected cache from future attention and repos to adjust retained keys’ position encodings.
- The default policy is a tool-response rolling-drop that retains the KV states of the latest 12 tool responses.
- This policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7% compared with threshold-triggered summarization.
- With the AKV server, it achieves up to 1.2× the output throughput and a 1.5× speedup in P95 request latency.
- Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design.
Paper links
External research summaries. These are not HDATF publications or measured product results.