Abstract
Translating chart images into executable plotting scripts-referred to as thechart-to-code generation task-requires Multimodal Large Language Models (MLLMs)to perform fine-grained visual parsing, precise code synthesis, and robustcross-modal reasoning. However, this task is inherently under-constrained:multiple valid code implementations can produce the same visual chart, andevaluation must consider both code correctness and visual fidelity acrossdiverse dimensions. This makes it difficult to learn accurate and generalizablemappings through standard supervised fine-tuning. To address these challenges,we propose a dual preference-guided refinement framework that combines afeedback-driven, dual-modality reward mechanism with iterative preferencelearning. Our approach introduces a structured variant generation strategy anda visual reward model to efficiently produce high-quality, aspect-awarepreference pairs-making preference collection scalable and supervision moretargeted. These preferences are used in an offline reinforcement learning setupto optimize the model toward multi-dimensional fidelity. Experimental resultsshow that our framework significantly enhances the performance ofgeneral-purpose open-source MLLMs, enabling them to generate high-qualityplotting code that rivals specialized chart-centric models and even someproprietary systems. The code and datasets are publicly available athttps://github.com/Zhihan72/Chart2Code.