Joint Representation Learning for Text and 3D Point Cloud

  • 2023-01-18 15:02:07
  • Rui Huang, Xuran Pan, Henry Zheng, Haojun Jiang, Zhifeng Xie, Shiji Song, Gao Huang
  • 1


Recent advancements in vision-language pre-training (e.g. CLIP) have shownthat vision models can benefit from language supervision. While many modelsusing language modality have achieved great success on 2D vision tasks, thejoint representation learning of 3D point cloud with text remainsunder-explored due to the difficulty of 3D-Text data pair acquisition and theirregularity of 3D data structure. In this paper, we propose a novel Text4Pointframework to construct language-guided 3D point cloud models. The key idea isutilizing 2D images as a bridge to connect the point cloud and the languagemodalities. The proposed Text4Point follows the pre-training and fine-tuningparadigm. During the pre-training stage, we establish the correspondence ofimages and point clouds based on the readily available RGB-D data and usecontrastive learning to align the image and point cloud representations.Together with the well-aligned image and text features achieved by CLIP, thepoint cloud features are implicitly aligned with the text embeddings. Further,we propose a Text Querying Module to integrate language information into 3Drepresentation learning by querying text embeddings with point cloud features.For fine-tuning, the model learns task-specific 3D representations underinformative language guidance from the label set without 2D images. Extensiveexperiments demonstrate that our model shows consistent improvement on variousdownstream tasks, such as point cloud semantic segmentation, instancesegmentation, and object detection. The code will be available here:


Quick Read (beta)

loading the full paper ...