Abstract
While multi-agent systems have been shown to significantly enhance theperformance of Large Language Models (LLMs) across various tasks andapplications, the dense interaction between scaling agents potentially hamperstheir efficiency and diversity. To address these challenges, we drawinspiration from the sparse mixture-of-agents (SMoE) and propose a sparsemixture-of-agents (SMoA) framework to improve the efficiency and diversity ofmulti-agent LLMs. Unlike completely connected structures, SMoA introduces novelResponse Selection and Early Stopping mechanisms to sparsify information flowsamong individual LLM agents, striking a balance between performance andefficiency. Additionally, inspired by the expert diversity principle in SMoEframeworks for workload balance between experts, we assign distinct roledescriptions to each LLM agent, fostering diverse and divergent thinking.Extensive experiments on reasoning, alignment, and fairness benchmarksdemonstrate that SMoA achieves performance comparable to traditionalmixture-of-agents approaches but with significantly lower computational costs.Further analysis reveals that SMoA is more stable, has a greater capacity toscale, and offers considerable potential through hyper-parameter optimization.Code and data will be available at: https://github.com/David-Li0406/SMoA.