Abstract
Depth map enhancement using paired high-resolution RGB images offers acost-effective solution for improving low-resolution depth data fromlightweight ToF sensors. Nevertheless, naively adopting a depth estimationpipeline to fuse the two modalities requires groundtruth depth maps forsupervision. To address this, we propose a self-supervised learning framework,SelfToF, which generates detailed and scale-aware depth maps. Starting from animage-based self-supervised depth estimation pipeline, we add low-resolutiondepth as inputs, design a new depth consistency loss, propose a scale-recoverymodule, and finally obtain a large performance boost. Furthermore, since theToF signal sparsity varies in real-world applications, we upgrade SelfToF toSelfToF* with submanifold convolution and guided feature fusion. Consequently,SelfToF* maintain robust performance across varying sparsity levels in ToFdata. Overall, our proposed method is both efficient and effective, as verifiedby extensive experiments on the NYU and ScanNet datasets. The code is availableat\href{https://github.com/denyingmxd/selftof}{https://github.com/denyingmxd/selftof}.