Abstract
While question-answering~(QA) benchmark performance is an automatic andscalable method to compare LLMs, it is an indirect method of evaluating theirunderlying problem-solving capabilities. Therefore, we propose a holistic andgeneralizable framework based on \emph{cascaded question disclosure} thatprovides a more accurate estimate of the models' problem-solving capabilitieswhile maintaining the scalability and automation. This approach collects modelresponses in a stagewise manner with each stage revealing partial informationabout the question designed to elicit generalized reasoning in LLMs. We findthat our approach not only provides a better comparison between LLMs, but alsoinduces better intermediate traces in models compared to the standard QAparadigm. We empirically verify this behavior on diverse reasoning andknowledge-heavy QA datasets by comparing LLMs of varying sizes and families.Our approach narrows the performance gap observed in the standard QA evaluationsettings, indicating that the prevalent indirect QA paradigm of evaluationoverestimates the differences in performance between models. We furthervalidate our findings by extensive ablation studies.