Abstract
The rapid advancement of visual generative models necessitates efficient andreliable evaluation methods. Arena platform, which gathers user votes on modelcomparisons, can rank models with human preferences. However, traditional Arenamethods, while established, require an excessive number of comparisons forranking to converge and are vulnerable to preference noise in voting,suggesting the need for better approaches tailored to contemporary evaluationchallenges. In this paper, we introduce K-Sort Arena, an efficient and reliableplatform based on a key insight: images and videos possess higher perceptualintuitiveness than texts, enabling rapid evaluation of multiple samplessimultaneously. Consequently, K-Sort Arena employs K-wise comparisons, allowingK models to engage in free-for-all competitions, which yield much richerinformation than pairwise comparisons. To enhance the robustness of the system,we leverage probabilistic modeling and Bayesian updating techniques. We proposean exploration-exploitation-based matchmaking strategy to facilitate moreinformative comparisons. In our experiments, K-Sort Arena exhibits 16.3x fasterconvergence compared to the widely used ELO algorithm. To further validate thesuperiority and obtain a comprehensive leaderboard, we collect human feedbackvia crowdsourced evaluations of numerous cutting-edge text-to-image andtext-to-video models. Thanks to its high efficiency, K-Sort Arena cancontinuously incorporate emerging models and update the leaderboard withminimal votes. Our project has undergone several months of internal testing andis now available at https://huggingface.co/spaces/ksort/K-Sort-Arena