Abstract
Timely and personalized treatment decisions are essential across a wide rangeof healthcare settings where patient responses can vary significantly andevolve over time. Clinical data used to support these treatment decisions areoften irregularly sampled, where missing data frequencies may implicitly conveyinformation about the patient's condition. Existing Reinforcement Learning (RL)based clinical decision support systems often ignore the missing patterns anddistort them with coarse discretization and simple imputation. They are alsopredominantly model-free and largely depend on retrospective data, which couldlead to insufficient exploration and bias by historical behaviors. To addressthese limitations, we propose medDreamer, a novel model-based reinforcementlearning framework for personalized treatment recommendation. medDreamercontains a world model with an Adaptive Feature Integration module thatsimulates latent patient states from irregular data and a two-phase policytrained on a hybrid of real and imagined trajectories. This enables learningoptimal policies that go beyond the sub-optimality of historical clinicaldecisions, while remaining close to real clinical data. We evaluate medDreameron both sepsis and mechanical ventilation treatment tasks using two large-scaleElectronic Health Records (EHRs) datasets. Comprehensive evaluations show thatmedDreamer significantly outperforms model-free and model-based baselines inboth clinical outcomes and off-policy metrics.