When Does LeJEPA Learn a World Model?
- SIGReg and VICReg enforce whitening, meaning they make each dimension independent with equal variance, and both methods maintained very high identifiability across all dimensions.
- InfoNCE is a widely used contrastive learning method. It works well in low dimensions, but in high-dimensional spaces it became extremely sensitive to hyperparameter tuning and often failed to preserve structure recovery without careful adjustment.
- For data collection, if we want a robot to learn a high-quality representation of its environment, the data it 'dreams' during pretraining should be biased and goal-oriented, rather than close to isotropic random walks.