When Does LeJEPA Learn a World Model?
- Published
- Source
- arXiv
- Paper number
- 244
- Field
- Machine Learning
- arXiv ID
- 2605.26379
Key points
- SIGReg and VICReg enforce whitening, meaning they make each dimension independent with equal variance, and both methods maintained very high identifiability across all dimensions.
- InfoNCE is a widely used contrastive learning method. It works well in low dimensions, but in high-dimensional spaces it became extremely sensitive to hyperparameter tuning and often failed to preserve structure recovery without careful adjustment.
- For data collection, if we want a robot to learn a high-quality representation of its environment, the data it 'dreams' during pretraining should be biased and goal-oriented, rather than close to isotropic random walks.
Paper links
External research summaries. These are not HDATF publications or measured product results.