Abstract
Most language grounding models learn to select the referred object from apool of object proposals provided by a pre-trained detector. This objectproposal bottleneck is limiting because an utterance may refer to visualentities at various levels of granularity, such as the chair, the leg of achair, or the tip of the front leg of a chair, which may be missed by thedetector. Recently, MDETR introduced a language grounding model for 2D imagesthat do not have such a box proposal bottleneck; instead of selecting objectsfrom a proposal pool, it instead decodes the referenced object boxes directlyfrom image and language features and achieves big leaps in performance. Wepropose a language grounding model for 3D scenes built on MDETR, which we callBEAUTY-DETR, from bottom-up and top-down DETR. BEAUTY-DETR attends on anadditional object proposal pool computed bottom-up from a pre-trained detector.Yet it decodes referenced objects without selecting them from the pool. In thisway, it uses powerful object detectors to help ground language without beingrestricted by their misses. Second, BEAUTY-DETR augments supervision fromlanguage grounding annotations by configuring object detection annotations aslanguage prompts to be grounded in images. The proposed model sets a newstate-of-the-art across popular 3D language grounding benchmarks withsignificant performance gains over previous 3D approaches (12.6% on SR3D, 11.6%on NR3D and 6.3% on ScanRefer). It outperforms a straightforward MDETR for the3D point clouds method we implemented by 6.7% on SR3D, 11.8% on NR3D and 5% onthe ScanRefer benchmark. When applied to language grounding in 2D images, itperforms on par with MDETR. We ablate each of the design choices of the modeland quantify their contribution to performance. Code and checkpoints areavailable at https://github.com/nickgkan/beauty_detr.