Abstract
Search-augmented language models combine web search with Large LanguageModels (LLMs) to improve response groundedness and freshness. However,analyzing these systems remains challenging: existing datasets are limited inscale and narrow in scope, often constrained to static, single-turn,fact-checking questions. In this work, we introduce Search Arena, acrowd-sourced, large-scale, human-preference dataset of over 24,000 pairedmulti-turn user interactions with search-augmented LLMs. The dataset spansdiverse intents and languages, and contains full system traces with around12,000 human preference votes. Our analysis reveals that user preferences areinfluenced by the number of citations, even when the cited content does notdirectly support the attributed claims, uncovering a gap between perceived andactual credibility. Furthermore, user preferences vary across cited sources,revealing that community-driven platforms are generally preferred and staticencyclopedic sources are not always appropriate and reliable. To assessperformance across different settings, we conduct cross-arena analyses bytesting search-augmented LLMs in a general-purpose chat environment andconventional LLMs in search-intensive settings. We find that web search doesnot degrade and may even improve performance in non-search settings; however,the quality in search settings is significantly affected if solely relying onthe model's parametric knowledge. We open-sourced the dataset to support futureresearch in this direction. Our dataset and code are available at:https://github.com/lmarena/search-arena.