Abstract
Nearly all human work is collaborative; thus, the evaluation of real-worldNLP applications often requires multiple dimensions that align with diversehuman perspectives. As real human evaluator resources are often scarce andcostly, the emerging "LLM-as-a-judge" paradigm sheds light on a promisingapproach to leverage LLM agents to believably simulate human evaluators. Yet,to date, existing LLM-as-a-judge approaches face two limitations: personadescriptions of agents are often arbitrarily designed, and the frameworks arenot generalizable to other tasks. To address these challenges, we proposeMAJ-EVAL, a Multi-Agent-as-Judge evaluation framework that can automaticallyconstruct multiple evaluator personas with distinct dimensions from relevanttext documents (e.g., research papers), instantiate LLM agents with thepersonas, and engage in-group debates with multi-agents to Generatemulti-dimensional feedback. Our evaluation experiments in both the educationaland medical domains demonstrate that MAJ-EVAL can generate evaluation resultsthat better align with human experts' ratings compared with conventionalautomated evaluation metrics and existing LLM-as-a-judge methods.