MAIRA-1: A specialised large multimodal model for radiology report generation

  • 2024-02-09 18:16:56
  • Stephanie L. Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Mercy Ranjit, Anton Schwaighofer, Fernando Pérez-García, Valentina Salvatelli, Shaury Srivastav, Anja Thieme, Noel Codella, Matthew P. Lungren, Maria Teodora Wetscherek, Ozan Oktay, Javier Alvarez-Valle
  • 0


We present a radiology-specific multimodal model for the task for generatingradiological reports from chest X-rays (CXRs). Our work builds on the idea thatlarge language model(s) can be equipped with multimodal capabilities throughalignment with pre-trained vision encoders. On natural images, this has beenshown to allow multimodal models to gain image understanding and descriptioncapabilities. Our proposed model (MAIRA-1) leverages a CXR-specific imageencoder in conjunction with a fine-tuned large language model based onVicuna-7B, and text-based data augmentation, to produce reports withstate-of-the-art quality. In particular, MAIRA-1 significantly improves on theradiologist-aligned RadCliQ metric and across all lexical metrics considered.Manual review of model outputs demonstrates promising fluency and accuracy ofgenerated reports while uncovering failure modes not captured by existingevaluation practices. More information and resources can be found on theproject website:


Quick Read (beta)

loading the full paper ...