Abstract
Transparent objects are common in daily life, and understanding theirmulti-layer depth information -- perceiving both the transparent surface andthe objects behind it -- is crucial for real-world applications that interactwith transparent materials. In this paper, we introduce LayeredDepth, the firstdataset with multi-layer depth annotations, including a real-world benchmarkand a synthetic data generator, to support the task of multi-layer depthestimation. Our real-world benchmark consists of 1,500 images from diversescenes, and evaluating state-of-the-art depth estimation methods on it revealsthat they struggle with transparent objects. The synthetic data generator isfully procedural and capable of providing training data for this task with anunlimited variety of objects and scene compositions. Using this generator, wecreate a synthetic dataset with 15,300 images. Baseline models training solelyon this synthetic dataset produce good cross-domain multi-layer depthestimation. Fine-tuning state-of-the-art single-layer depth models on itsubstantially improves their performance on transparent objects, withquadruplet accuracy on our benchmark increased from 55.14% to 75.20%. Allimages and validation annotations are available under CC0 athttps://layereddepth.cs.princeton.edu.