Abstract
Current facial emotion recognition systems are predominately trained topredict a fixed set of predefined categories or abstract dimensional values.This constrained form of supervision hinders generalization and applicability,as it reduces the rich and nuanced spectrum of emotions into oversimplifiedlabels or scales. In contrast, natural language provides a more flexible,expressive, and interpretable way to represent emotions, offering a muchbroader source of supervision. Yet, leveraging semantically rich naturallanguage captions as supervisory signals for facial emotion representationlearning remains relatively underexplored, primarily due to two key challenges:1) the lack of large-scale caption datasets with rich emotional semantics, and2) the absence of effective frameworks tailored to harness such richsupervision. To this end, we introduce EmoCap100K, a large-scale facial emotioncaption dataset comprising over 100,000 samples, featuring rich and structuredsemantic descriptions that capture both global affective states andfine-grained local facial behaviors. Building upon this dataset, we furtherpropose EmoCapCLIP, which incorporates a joint global-local contrastivelearning framework enhanced by a cross-modal guided positive mining module.This design facilitates the comprehensive exploitation of multi-level captioninformation while accommodating semantic similarities between closely relatedexpressions. Extensive evaluations on over 20 benchmarks covering five tasksdemonstrate the superior performance of our method, highlighting the promise oflearning facial emotion representations from large-scale semantically richcaptions. The code and data will be available athttps://github.com/sunlicai/EmoCapCLIP.