Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
- Published
- Source
- arXiv
- Paper number
- 030
- Field
- Agents / Ensembling
- arXiv ID
- 2502.00674
Key points
- Traditional mixture-of-agents approaches assume that combining different LLMs leads to better performance.
- There is still limited understanding of the fundamental tradeoff between diversity and quality in LLM ensembles.
- The paper introduces Self-MoA, which generates multiple outputs from a single high-performing model.
- It also develops Self-MoA-Seq to handle more outputs under context-length constraints.
- It conducts a comprehensive analysis of the quality-diversity relationship across more than 200 experiments.
- Ensemble performance is more sensitive to output quality than to diversity.
Paper links
External research summaries. These are not HDATF publications or measured product results.