Deep Delta Learning
- Published
- Source
- arXiv
- Paper number
- 105
- Field
- Architecture / Learning
- arXiv ID
- 2601.00417
Key points
- Deep residual networks, or ResNets, impose a strictly additive inductive bias through identity shortcut connections, which limits the complexity of learnable feature transformations.
- The fixed identity Jacobian of ResNet shortcut connections constrains the network's ability to model complex, non-monotonic state transitions that require dynamics often characterized by negative eigenvalues, such as oscillation or adversarial behavior.
- Standard residual updates, which can be viewed as forward Euler steps, exhibit a strong translational bias and lack an explicit mechanism for feature forgetting or reorientation.
- The paper introduces the Delta Residual Block, which extends the usual additive residual update, where the next layer equals the current layer plus a transformed term, into a multiplicative form that multiplies the shortcut connection by a data-dependent geometric transform matrix and a key-value correction term.
- The key is the Delta Operator, in which A(X) = I − β(X) times the outer product of k(X) with itself. This rank-1 perturbation of the identity matrix applies a spatial geometric transformation.
- The operator is parameterized by a learnable, data-dependent reflection direction k(X) and a gating scalar beta(X) that is dynamically controlled in the range from 0 to 2.
Paper links
External research summaries. These are not HDATF publications or measured product results.