Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents
- Published
- Source
- arXiv
- Paper number
- 1208
- Field
- Agents / Memory
- arXiv ID
- 2610.18052
Key points
- Bemula is the first benchmark dedicated to memory unlearning in LLM agents, the ability to delete exactly what the user requests.
- Its tasks jointly measure removal of requested facts and preservation of the rest of memory.
- Across all evaluated systems and backbones, no system passed more than 1.5% of forget requests end to end, and most passed none at all.
- A human validation of the LLM judge on 100 sampled items yielded 98% agreement, supporting evaluation reliability.
Paper links
External research summaries. These are not HDATF publications or measured product results.