Abstract
The multimodal language models (MLMs) based on generative pre-trainedTransformer are considered powerful candidates for unifying various domains andtasks. MLMs developed for remote sensing (RS) have demonstrated outstandingperformance in multiple tasks, such as visual question answering and visualgrounding. In addition to visual grounding that detects specific objectscorresponded to given instruction, aerial detection, which detects all objectsof multiple categories, is also a valuable and challenging task for RSfoundation models. However, aerial detection has not been explored by existingRS MLMs because the autoregressive prediction mechanism of MLMs differssignificantly from the detection outputs. In this paper, we present a simplebaseline for applying MLMs to aerial detection for the first time, namedLMMRotate. Specifically, we first introduce a normalization method to transformdetection outputs into textual outputs to be compatible with the MLM framework.Then, we propose a evaluation method, which ensures a fair comparison betweenMLMs and conventional object detection models. We construct the baseline byfine-tuning open-source general-purpose MLMs and achieve impressive detectionperformance comparable to conventional detector. We hope that this baselinewill serve as a reference for future MLM development, enabling morecomprehensive capabilities for understanding RS images. Code is available athttps://github.com/Li-Qingyun/mllm-mmrotate.