Abstract
Foundation models hold significant promise in healthcare, given theircapacity to extract meaningful representations independent of downstream tasks.This property has enabled state-of-the-art performance across several clinicalapplications trained on structured electronic health record (EHR) data, even insettings with limited labeled data, a prevalent challenge in healthcare.However, there is little consensus on these models' potential for clinicalutility due to the lack of desiderata of comprehensive and meaningful tasks andsufficiently diverse evaluations to characterize the benefit over conventionalsupervised learning. To address this gap, we propose a suite of clinicallymeaningful tasks spanning patient outcomes, early prediction of acute andchronic conditions, including desiderata for robust evaluations. We evaluatestate-of-the-art foundation models on EHR data consisting of 5 million patientsfrom Columbia University Irving Medical Center (CUMC), a large urban academicmedical center in New York City, across 14 clinically relevant tasks. Wemeasure overall accuracy, calibration, and subpopulation performance to surfacetradeoffs based on the choice of pre-training, tokenization, and datarepresentation strategies. Our study aims to advance the empirical evaluationof structured EHR foundation models and guide the development of futurehealthcare foundation models.