Abstract
Grounding natural language queries in graphical user interfaces (GUIs) posesunique challenges due to the diversity of visual elements, spatial clutter, andthe ambiguity of language. In this paper, we introduce DiMo-GUI, atraining-free framework for GUI grounding that leverages two core strategies:dynamic visual grounding and modality-aware optimization. Instead of treatingthe GUI as a monolithic image, our method splits the input into textualelements and iconic elements, allowing the model to reason over each modalityindependently using general-purpose vision-language models. When predictionsare ambiguous or incorrect, DiMo-GUI dynamically focuses attention bygenerating candidate focal regions centered on the model's initial predictionsand incrementally zooms into subregions to refine the grounding result. Thishierarchical refinement process helps disambiguate visually crowded layoutswithout the need for additional training or annotations. We evaluate ourapproach on standard GUI grounding benchmarks and demonstrate consistentimprovements over baseline inference pipelines, highlighting the effectivenessof combining modality separation with region-focused reasoning.