Abstract
We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks formultisensory object-centric learning, centered around object recognition,reconstruction, and manipulation with sight, sound, and touch. We alsointroduce the ObjectFolder Real dataset, including the multisensorymeasurements for 100 real-world household objects, building upon a newlydesigned pipeline for collecting the 3D meshes, videos, impact sounds, andtactile readings of real-world objects. We conduct systematic benchmarking onboth the 1,000 multisensory neural objects from ObjectFolder, and the realmultisensory data from ObjectFolder Real. Our results demonstrate theimportance of multisensory perception and reveal the respective roles ofvision, audio, and touch for different object-centric learning tasks. Bypublicly releasing our dataset and benchmark suite, we hope to catalyze andenable new research in multisensory object-centric learning in computer vision,robotics, and beyond. Project page: https://objectfolder.stanford.edu