Abstract
ARC Challenge appears more difficult than ARC Easy for modern LLMs primarilydue to an evaluation setup that prevents direct comparison of answer choicesrather than inherent complexity. Although some researchers have quietly shiftedto a more appropriate scheme over the last year, the implications of thischange have yet to be widely acknowledged. We highlight this overlooked shift,show how similar evaluation practices falsely imply reasoning deficits in otherbenchmarks, and demonstrate that fairer methods dramatically reduce performancegaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doingso, we reveal how evaluation shapes perceived difficulty and offer guidelinesto ensure that multiple-choice evaluations accurately reflect actual modelcapabilities.