Abstract
Large multimodal models (LMMs) now excel on many vision language benchmarks,however, they still struggle with human centered criteria such as fairness,ethics, empathy, and inclusivity, key to aligning with human values. Weintroduce HumaniBench, a holistic benchmark of 32K real-world image questionpairs, annotated via a scalable GPT4o assisted pipeline and exhaustivelyverified by domain experts. HumaniBench evaluates seven Human Centered AI(HCAI) principles: fairness, ethics, understanding, reasoning, languageinclusivity, empathy, and robustness, across seven diverse tasks, includingopen and closed ended visual question answering (VQA), multilingual QA, visualgrounding, empathetic captioning, and robustness tests. Benchmarking 15 stateof the art LMMs (open and closed source) reveals that proprietary modelsgenerally lead, though robustness and visual grounding remain weak points. Someopen-source models also struggle to balance accuracy with adherence tohuman-aligned principles. HumaniBench is the first benchmark purpose builtaround HCAI principles. It provides a rigorous testbed for diagnosing alignmentgaps and guiding LMMs toward behavior that is both accurate and sociallyresponsible. Dataset, annotation prompts, and evaluation code are available at:https://vectorinstitute.github.io/HumaniBench