When can an interpretability claim be trusted? Stability and certification of feature dimensionality, ridge leverage and the feature recovery rate

Published
Source
arXiv
Paper number
1489
Field
Interpretability
arXiv ID
2611.16190

Key points

  • Small changes to the model weights, a different training seed, or a different threshold can reverse them. In earlier work in this program, a norm threshold alone produced a 24.8% apparent failure of a well-known regularity.
  • We characterize exactly where feature dimensionality is discontinuous: where a feature’s weight vector vanishes while some direction is orthogonal to all the other feature vectors.
  • We replace feature dimensionality with a regularized (ridge) version and prove sharp stability bounds.
  • We also prove certificates that state when a claim “feature i is recovered” must survive a given change of the dictionary.
  • Our experiments use toy models, synthetic dictionaries and small dictionaries trained with three seeds; we have not tested real language models.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)