Merlin: A Vision Language Foundation Model for 3D Computed Tomography

  • 2024-06-10 18:53:01
  • Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, Christian Bluethgen, Malte Engmann Kjeldskov Jensen, Sophie Ostmeier, Maya Varma, Jeya Maria Jose Valanarasu, Zhongnan Fang, Zepeng Huo, Zaid Nabulsi, Diego Ardila, Wei-Hung Weng, Edson Amaro Junior, Neera Ahuja, Jason Fries, Nigam H. Shah, Andrew Johnston, Robert D. Boutin, Andrew Wentland, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, Akshay S. Chaudhari
  • 0


Over 85 million computed tomography (CT) scans are performed annually in theUS, of which approximately one quarter focus on the abdomen. Given the currentradiologist shortage, there is a large impetus to use artificial intelligenceto alleviate the burden of interpreting these complex imaging studies. Priorstate-of-the-art approaches for automated medical image interpretation leveragevision language models (VLMs). However, current medical VLMs are generallylimited to 2D images and short reports, and do not leverage electronic healthrecord (EHR) data for supervision. We introduce Merlin - a 3D VLM that we trainusing paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes(1.8+ million codes), and radiology reports (6+ million tokens). We evaluateMerlin on 6 task types and 752 individual tasks. The non-adapted(off-the-shelf) tasks include zero-shot findings classification (31 findings),phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval(image to findings and image to impressions), while model adapted tasks include5-year disease prediction (6 diseases), radiology report generation, and 3Dsemantic segmentation (20 organs). We perform internal validation on a test setof 5,137 CTs, and external validation on 7,000 clinical CTs and on two publicCT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevantevaluations, we assess the efficacy of various network architectures andtraining strategies to depict that Merlin has favorable performance to existingtask-specific baselines. We derive data scaling laws to empirically assesstraining data needs for requisite downstream task performance. Furthermore,unlike conventional VLMs that require hundreds of GPUs for training, we performall training on a single GPU.


Quick Read (beta)

loading the full paper ...