Abstract
Target volume contouring for radiation therapy is considered significantlymore challenging than the normal organ segmentation tasks as it necessitatesthe utilization of both image and text-based clinical information. Inspired bythe recent advancement of large language models (LLMs) that can facilitate theintegration of the textural information and images, here we present a novelLLM-driven multi-modal AI that utilizes the clinical text information and isapplicable to the challenging task of target volume contouring for radiationtherapy, and validate it within the context of breast cancer radiation therapytarget volume contouring. Using external validation and data-insufficientenvironments, which attributes highly conducive to real-world applications, wedemonstrate that the proposed model exhibits markedly improved performancecompared to conventional vision-only AI models, particularly exhibiting robustgeneralization performance and data-efficiency. To our best knowledge, this isthe first LLM-driven multimodal AI model that integrates the clinical textinformation into target volume delineation for radiation oncology.