Abstract
Remote estimation of vital signs enables health monitoring for situations inwhich contact-based devices are either not available, too intrusive, or tooexpensive. In this paper, we present a modular, interpretable pipeline forpulse signal estimation from video of the face that achieves state-of-the-artresults on publicly available datasets.Our imaging photoplethysmography (iPPG)system consists of three modules: face and landmark detection, time-seriesextraction, and pulse signal/pulse rate estimation. Unlike many deep learningmethods that make use of a single black-box model that maps directly from inputvideo to output signal or heart rate, our modular approach enables each of thethree parts of the pipeline to be interpreted individually. The pulse signalestimation module, which we call TURNIP (Time-Series U-Net with Recurrence forNoise-Robust Imaging Photoplethysmography), allows the system to faithfullyreconstruct the underlying pulse signal waveform and uses it to measure heartrate and pulse rate variability metrics, even in the presence of motion. Whenparts of the face are occluded due to extreme head poses, our system explicitlydetects such "self-occluded" regions and maintains estimation robustnessdespite the missing information. Our algorithm provides reliable heart rateestimates without the need for specialized sensors or contact with the skin,outperforming previous iPPG methods on both color (RGB) and near-infrared (NIR)datasets.