K2-Horizon: A Frontier Fleet of Open Source Language Models

Published
Source
arXiv
Paper number
1491
Field
LLMs / NLP
arXiv ID
2611.21003

Key points

  • The researchers released six fully open language models from a 0.9B dense model to a 375B-A23B MoE, for general-purpose reasoning and agentic work across math and science, software engineering, and deep research.
  • The authors introduced MoVA, applying MoE to attention value projections, and softplus attention gating; the 36B-A4B model nearly matches a 32B dense model trained on the same data with 1/7 the active parameters.
  • Uno is a lossless diffusion adapter that speeds up decoding by up to 2.2× without a separate draft model.
  • The released artifacts include intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code; the pipelines produced roughly 10T synthetic pre-training tokens.
  • Four of the models share one 22T-token pre-training corpus and recipe, and the 7B and 3.7B lead SWE-bench Verified by wide margins.

Paper links

External research summaries. These are not HDATF publications or measured product results.

Read original (opens in a new tab)