Abstract
Medical image registration is an essential topic in medical image analysis.In this paper, we propose a method for medical image registration using apretrained large language model. We find that using the pretrained largelanguage model to encode deep features of the medical images in theregistration model can effectively improve image registration accuracy,indicating the great potential of the large language model in medical imageregistration tasks. We use dual encoders to perform deep feature extraction onimage pairs and then input the features into the pretrained large languagemodel. To adapt the large language model to our registration task, the weightsof the large language model are frozen in the registration model, and anadapter is utilized to fine-tune the large language model, which aims at (a)mapping the visual tokens to the language space before the large language modelcomputing, (b) project the modeled language tokens output from the largelanguage model to the visual space. Our method combines output features fromthe fine-tuned large language model with the features output from each encoderlayer to gradually generate the deformation fields required for registration inthe decoder. To demonstrate the effectiveness of the large prediction model inregistration tasks, we conducted experiments on knee and brain MRI and achievedstate-of-the-art results.