Abstract
The colorectal polyps classification is a critical clinical examination. Toimprove the classification accuracy, most computer-aided diagnosis algorithmsrecognize colorectal polyps by adopting Narrow-Band Imaging (NBI). However, theNBI usually suffers from missing utilization in real clinic scenarios since theacquisition of this specific image requires manual switching of the light modewhen polyps have been detected by using White-Light (WL) images. To avoid theabove situation, we propose a novel method to directly achieve accuratewhite-light colonoscopy image classification by conducting structuredcross-modal representation consistency. In practice, a pair of multi-modalimages, i.e. NBI and WL, are fed into a shared Transformer to extracthierarchical feature representations. Then a novel designed Spatial AttentionModule (SAM) is adopted to calculate the similarities between the class tokenand patch tokens %from multi-levels for a specific modality image. By aligningthe class tokens and spatial attention maps of paired NBI and WL images atdifferent levels, the Transformer achieves the ability to keep both global andlocal representation consistency for the above two modalities. Extensiveexperimental results illustrate the proposed method outperforms the recentstudies with a margin, realizing multi-modal prediction with a singleTransformer while greatly improving the classification accuracy when only withWL images.