Abstract
The forecasting of disease progression from images is a holy grail forclinical decision making. However, this task is complicated by the inherenthigh dimensionality, temporal sparsity and sampling irregularity inlongitudinal image acquisitions. Existing methods often rely on extractinghand-crafted features and performing time-series analysis in this vector space,leading to a loss of rich spatial information within the images. To overcomethese challenges, we introduce ImageFlowNet, a novel framework that learnslatent-space flow fields that evolve multiscale representations in jointembedding spaces using neural ODEs and SDEs to model disease progression in theimage domain. Notably, ImageFlowNet learns multiscale joint representationspaces by combining cohorts of patients together so that information can betransferred between the patient samples. The dynamics then provide plausibletrajectories of progression, with the SDE providing alternative trajectoriesfrom the same starting point. We provide theoretical insights that support ourformulation of ODEs, and motivate our regularizations involving high-levelvisual features, latent space organization, and trajectory smoothness. We thendemonstrate ImageFlowNet's effectiveness through empirical evaluations on threelongitudinal medical image datasets depicting progression in retinal geographicatrophy, multiple sclerosis, and glioblastoma.