Algebraic Decomposition Theory for Transformer Length Generalization
- Published
- Source
- arXiv
- Paper number
- 909
- Field
- AI / General
- arXiv ID
- 2608.13433
Key points
- Two problems that look structurally almost identical can diverge into cases that do and do not generalize in length, and prior theory could not explain the difference; this paper gives the first complete decision criterion.
- The decision algorithm runs in polynomial time, so it is not just theoretical but also practically checkable.
- The key discovery is that unbounded counting is the seed of transformer length generalization, and the paper formalizes this through iterated wreath products of the integer-addition group.
- When GPT-2 is trained up to length 50 and tested up to length 500, languages inside C-RASP keep their accuracy while languages outside it collapse right after the training length.
- The work provides a theoretical foundation for understanding why structured outputs such as JSON and long agent state traces break down as they grow longer.
Paper links
External research summaries. These are not HDATF publications or measured product results.