There Will Be a Scientific Theory of Deep Learning

Key points

Traditional deep learning theory struggles to explain the complex, nonconvex, and overparameterized behavior seen in modern neural networks.
Current deep learning system design and optimization practices often rely on empirical trial and error rather than a principled scientific foundation.
A fundamental scientific understanding is needed to describe, characterize, and control increasingly powerful AI systems in terms of safety and interpretability.
The paper proposes a scientific theory of learning dynamics for deep learning that aims to characterize the fundamental properties of neural networks and their training processes through first-principles computation.
It advocates a scientific methodology for deep learning theory that emphasizes empirical observation, falsifiable predictions, and the search for simple, unifying principles.
It synthesizes and reinterprets prior work across five categories, such as analytically solvable settings, insightful limits, and empirical laws, as evidence for the theory's feasibility and emergence.

External research summaries. These are not HDATF publications or measured product results.