Abstract
Structured image understanding, such as interpreting tables and charts,requires strategically refocusing across various structures and texts within animage, forming a reasoning sequence to arrive at the final answer. However,current multimodal large language models (LLMs) lack this multihop selectiveattention capability. In this work, we introduce ReFocus, a simple yeteffective framework that equips multimodal LLMs with the ability to generate"visual thoughts" by performing visual editing on the input image through code,shifting and refining their visual focuses. Specifically, ReFocus enablesmultimodal LLMs to generate Python codes to call tools and modify the inputimage, sequentially drawing boxes, highlighting sections, and masking outareas, thereby enhancing the visual reasoning process. We experiment upon awide range of structured image understanding tasks involving tables and charts.ReFocus largely improves performance on all tasks over GPT-4o without visualediting, yielding an average gain of 11.0% on table tasks and 6.8% on charttasks. We present an in-depth analysis of the effects of different visualedits, and reasons why ReFocus can improve the performance without introducingadditional information. Further, we collect a 14k training set using ReFocus,and prove that such visual chain-of-thought with intermediate informationoffers a better supervision than standard VQA data, reaching a 8.0% averagegain over the same model trained with QA pairs and 2.6% over CoT.