Abstract
Recent advancements in multimodal pre-training methods have shown promisingefficacy in 3D representation learning by aligning features across 3D modality,their 2D counterpart modality, and corresponding language modality. However,the methods used by existing multimodal pre-training frameworks to gathermultimodal data for 3D applications lack scalability and comprehensiveness,potentially constraining the full potential of multimodal learning. The mainbottleneck lies in the language modality's scalability and comprehensiveness.To address this bottleneck, we introduce ULIP-2, a multimodal pre-trainingframework that leverages state-of-the-art multimodal large language models(LLMs) pre-trained on extensive knowledge to automatically generate holisticlanguage counterparts for 3D objects. We conduct experiments on two large-scaledatasets, Objaverse and ShapeNet55, and release our generated three-modalitytriplet datasets (3D Point Cloud - Image - Language), named "ULIP-ObjaverseTriplets" and "ULIP-ShapeNet Triplets". ULIP-2 requires only 3D data itself andeliminates the need for any manual annotation effort, demonstrating itsscalability; and ULIP-2 achieves remarkable improvements on downstreamzero-shot classification on ModelNet40 (74% Top1 Accuracy). Moreover, ULIP-2sets a new record on the real-world ScanObjectNN benchmark (91.5% OverallAccuracy) while utilizing only 1.4 million parameters(~10x fewer than currentSOTA), signifying a breakthrough in scalable multimodal 3D representationlearning without human annotations. The code and datasets are available athttps://github.com/salesforce/ULIP.