Titans: Learning to Memorize at Test Time
- Published
- Source
- arXiv
- Paper number
- 019
- Field
- Architecture / Memory
- arXiv ID
- 2501.00663
Key points
- Existing models struggle to process and remember very long sequences effectively.
- Transformers have quadratic complexity, which limits the context window.
- Current approaches lack an effective way to combine different memory types, such as short-term, long-term, and persistent memory.
- Models face challenges in generalization and length extrapolation.
- It introduces a neural long-term memory module that learns to remember at test time based on a surprise signal.
- It presents three architectural variants for integrating memory effectively: Memory as Context, Memory as Gate, and Memory as Layer.
Paper links
External research summaries. These are not HDATF publications or measured product results.