Decoding Looped Transformers Better for (Almost) Free
- Published
- Source
- arXiv
- Paper number
- 1149
- Field
- Machine Learning
- arXiv ID
- 2610.02185
Key points
- They proposed training-free contrastive decoding that uses the first recurrent pass of looped transformers as a weak model.
- LoopCD-Logits raised Ouro-2.6B's AIME 2024 pass@1 from 61.88% to 73.33%, and LoopCD-Hidden lifted Huginn's HumanEval from 22.56% to 31.71%.
- Halving the number of recurrent loops still matched or exceeded full-depth baselines, cutting compute by 22.5% to 48.2%.
- They showed that multiple-choice tasks need strong guidance (omega=0.5) while long generation tasks need weak guidance (omega=0.2-0.3).
Paper links
External research summaries. These are not HDATF publications or measured product results.