Abstract
Colorectal cancer (CRC) grading is a critical prognostic factor but remainshampered by inter-observer variability and the privacy constraints ofmulti-institutional data sharing. While deep learning offers a path toautomation, centralized training models conflict with data governanceregulations and neglect the diagnostic importance of multi-scale analysis. Inthis work, we propose a scalable, privacy-preserving federated learning (FL)framework for CRC histopathological grading that integrates multi-scale featurelearning within a distributed training paradigm. Our approach employs adual-stream ResNetRS50 backbone to concurrently capture fine-grained nucleardetail and broader tissue-level context. This architecture is integrated into arobust FL system stabilized using FedProx to mitigate client drift acrossheterogeneous data distributions from multiple hospitals. Extensive evaluationon the CRC-HGD dataset demonstrates that our framework achieves an overallaccuracy of 83.5%, outperforming a comparable centralized model (81.6%).Crucially, the system excels in identifying the most aggressive Grade IIItumors with a high recall of 87.5%, a key clinical priority to preventdangerous false negatives. Performance further improves with highermagnification, reaching 88.0% accuracy at 40x. These results validate that ourfederated multi-scale approach not only preserves patient privacy but alsoenhances model performance and generalization. The proposed modular pipeline,with built-in preprocessing, checkpointing, and error handling, establishes afoundational step toward deployable, privacy-aware clinical AI for digitalpathology.