K2-Horizon: A Frontier Fleet of Open Source Language Models
- Published
- Source
- arXiv
- Paper number
- 1491
- Field
- LLMs / NLP
- arXiv ID
- 2611.21003
Key points
- The researchers released six fully open language models from a 0.9B dense model to a 375B-A23B MoE, for general-purpose reasoning and agentic work across math and science, software engineering, and deep research.
- The authors introduced MoVA, applying MoE to attention value projections, and softplus attention gating; the 36B-A4B model nearly matches a 32B dense model trained on the same data with 1/7 the active parameters.
- Uno is a lossless diffusion adapter that speeds up decoding by up to 2.2× without a separate draft model.
- The released artifacts include intermediate checkpoints from every training stage, pre-training data, training logs, and all training, evaluation, and infrastructure code; the pipelines produced roughly 10T synthetic pre-training tokens.
- Four of the models share one 22T-token pre-training corpus and recipe, and the 7B and 3.7B lead SWE-bench Verified by wide margins.
Paper links
External research summaries. These are not HDATF publications or measured product results.