Abstract
The glyphic writing system of Chinese incorporates information-rich visualfeatures in each character, such as radicals that provide hints about meaningor pronunciation. However, there has been no investigation into whethercontemporary Large Language Models (LLMs) and Vision-Language Models (VLMs) canharness these sub-character features in Chinese through prompting. In thisstudy, we establish a benchmark to evaluate LLMs' and VLMs' understanding ofvisual elements in Chinese characters, including radicals, compositionstructures, strokes, and stroke counts. Our results reveal that modelssurprisingly exhibit some, but still limited, knowledge of the visualinformation, regardless of whether images of characters are provided. To incitemodels' ability to use radicals, we further experiment with incorporatingradicals into the prompts for Chinese language understanding tasks. We observeconsistent improvement in Part-Of-Speech tagging when providing additionalinformation about radicals, suggesting the potential to enhance CLP byintegrating sub-character information.