Abstract
The Automated Audio Captioning (AAC) task asks models to generate naturallanguage descriptions of an audio input. Evaluating these machine-generatedaudio captions is a complex task that requires considering diverse factors,among them, auditory scene understanding, sound-object inference, temporalcoherence, and the environmental context of the scene. While current methodsfocus on specific aspects, they often fail to provide an overall score thataligns well with human judgment. In this work, we propose CLAIR-A, a simple andflexible method that leverages the zero-shot capabilities of large languagemodels (LLMs) to evaluate candidate audio captions by directly asking LLMs fora semantic distance score. In our evaluations, CLAIR-A better predicts humanjudgements of quality compared to traditional metrics, with a 5.8% relativeaccuracy improvement compared to the domain-specific FENSE metric and up to 11%over the best general-purpose measure on the Clotho-Eval dataset. Moreover,CLAIR-A offers more transparency by allowing the language model to explain thereasoning behind its scores, with these explanations rated up to 30% better byhuman evaluators than those provided by baseline methods. CLAIR-A is madepublicly available at https://github.com/DavidMChan/clair-a.