PaLI: A Jointly-Scaled Multilingual Language-Image Model

  • 2022-09-14 18:24:07
  • Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut
  • 23

Abstract

Effective scaling and a flexible task interface enable large language modelsto excel at many tasks.PaLI(PathwaysLanguage andImage model) extends thisapproach to the joint modeling of language and vision. PaLI generates textbased on visual and textual inputs, and with this interface performs manyvision, language, and multimodal tasks, in many languages. To train PaLI, wemake use of large pretrained encoder-decoder language models and VisionTransformers (ViTs). This allows us to capitalize on their existingcapabilities and leverage the substantial cost of training them. We find thatjoint scaling of the vision and language components is important. Sinceexisting Transformers for language are much larger than their visioncounterparts, we train the largest ViT to date (ViT-e) to quantify the benefitsfrom even larger-capacity vision models. To train PaLI, we create a largemultilingual mix of pretraining tasks, based on a new image-text training setcontaining 10B images and texts in over 100 languages. PaLI achievesstate-of-the-art in multiple vision and language tasks (such as captioning,visual question-answering, scene-text understanding), while retaining a simple,modular, and scalable design.

 

Quick Read (beta)

loading the full paper ...