Abstract
Natural language is a powerful complementary modality of communication fordata visualizations, such as bar and line charts. To facilitate chart-basedreasoning using natural language, various downstream tasks have been introducedrecently such as chart question answering, chart summarization, andfact-checking with charts. These tasks pose a unique challenge, demanding bothvision-language reasoning and a nuanced understanding of chart data tables,visual encodings, and natural language prompts. Despite the recent success ofLarge Language Models (LLMs) across diverse NLP tasks, their abilities andlimitations in the realm of data visualization remain under-explored, possiblydue to their lack of multi-modal capabilities. To bridge the gap, this paperpresents the first comprehensive evaluation of the recently developed largevision language models (LVLMs) for chart understanding and reasoning tasks. Ourevaluation includes a comprehensive assessment of LVLMs, including GPT-4V andGemini, across four major chart reasoning tasks. Furthermore, we perform aqualitative evaluation of LVLMs' performance on a diverse range of charts,aiming to provide a thorough analysis of their strengths and weaknesses. Ourfindings reveal that LVLMs demonstrate impressive abilities in generatingfluent texts covering high-level data insights while also encountering commonproblems like hallucinations, factual errors, and data bias. We highlight thekey strengths and limitations of chart comprehension tasks, offering insightsfor future research.