Abstract
In arena-style evaluation of large language models (LLMs), two LLMs respondto a user query, and the user chooses the winning response or deems the"battle" a draw, resulting in an adjustment to the ratings of both models. Theprevailing approach for modeling these rating dynamics is to view battles astwo-player game matches, as in chess, and apply the Elo rating system and itsderivatives. In this paper, we critically examine this paradigm. Specifically,we question whether a draw genuinely means that the two models are equal andhence whether their ratings should be equalized. Instead, we conjecture thatdraws are more indicative of query difficulty: if the query is too easy, thenboth models are more likely to succeed equally. On three real-world arenadatasets, we show that ignoring rating updates for draws yields a 1-3% relativeincrease in battle outcome prediction accuracy (which includes draws) for allfour rating systems studied. Further analyses suggest that draws occur more forqueries rated as very easy and those as highly objective, with risk ratios of1.37 and 1.35, respectively. We recommend future rating systems to reconsiderexisting draw semantics and to account for query properties in rating updates.