Abstract
Although the Segment Anything Model (SAM) has demonstrated impressiveperformance in 2D natural image segmentation, its application to 3D volumetricmedical images reveals significant shortcomings, namely suboptimal performanceand unstable prediction, necessitating an excessive number of prompt points toattain the desired outcomes. These issues can hardly be addressed byfine-tuning SAM on medical data because the original 2D structure of SAMneglects 3D spatial information. In this paper, we introduce SAM-Med3D, themost comprehensive study to modify SAM for 3D medical images. Our approach ischaracterized by its comprehensiveness in two primary aspects: firstly, bycomprehensively reformulating SAM to a thorough 3D architecture trained on acomprehensively processed large-scale volumetric medical dataset; and secondly,by providing a comprehensive evaluation of its performance. Specifically, wetrain SAM-Med3D with over 131K 3D masks and 247 categories. Our SAM-Med3Dexcels at capturing 3D spatial information, exhibiting competitive performancewith significantly fewer prompt points than the top-performing fine-tuned SAMin the medical domain. We then evaluate its capabilities across 15 datasets andanalyze it from multiple perspectives, including anatomical structures,modalities, targets, and generalization abilities. Our approach, compared withSAM, showcases pronouncedly enhanced efficiency and broad segmentationcapabilities for 3D volumetric medical images. Our code is released athttps://github.com/uni-medical/SAM-Med3D.