Symmetry in language statistics shapes the geometry of model representations

Key points

There has long been no clear and unified theoretical explanation for the geometric patterns, such as circles and one-dimensional manifolds, that are widely observed in language model representations.
There is no comprehensive organizing principle that explains the apparent universality of these geometric structures across different LLM architectures and tasks.
Existing theoretical work has not sufficiently explained how data distributions and unsupervised learning settings induce these specific representation geometries.
The paper proposes that translational symmetry in pairwise word co-occurrence statistics acts as a universal organizing principle for representation geometry.
From this symmetry, it develops a mathematical theory that analytically derives specific manifold geometries, such as circles and wavy one-dimensional manifolds.
It empirically validates these theoretical predictions using both simple word embedding models and complex Transformer-based large language models.

External research summaries. These are not HDATF publications or measured product results.