Abstract
Recent 3D retrieval systems are typically designed for simple, controlledscenarios, such as identifying an object from a cropped image or a briefdescription. However, real-world scenarios are more complex, often requiringthe recognition of an object in a cluttered scene based on a vague, free-formdescription. To this end, we present ROOMELSA, a new benchmark designed toevaluate a system's ability to interpret natural language. Specifically,ROOMELSA attends to a specific region within a panoramic room image andaccurately retrieves the corresponding 3D model from a large database. Inaddition, ROOMELSA includes over 1,600 apartment scenes, nearly 5,200 rooms,and more than 44,000 targeted queries. Empirically, while coarse objectretrieval is largely solved, only one top-performing model consistently rankedthe correct match first across nearly all test cases. Notably, a lightweightCLIP-based model also performed well, although it struggled with subtlevariations in materials, part structures, and contextual cues, resulting inoccasional errors. These findings highlight the importance of tightlyintegrating visual and language understanding. By bridging the gap betweenscene-level grounding and fine-grained 3D retrieval, ROOMELSA establishes a newbenchmark for advancing robust, real-world 3D recognition systems.