Abstract
Effective scaling and a flexible task interface enable large language modelsto excel at many tasks. We present PaLI (Pathways Language and Image model), amodel that extends this approach to the joint modeling of language and vision.PaLI generates text based on visual and textual inputs, and with this interfaceperforms many vision, language, and multimodal tasks, in many languages. Totrain PaLI, we make use of large pre-trained encoder-decoder language modelsand Vision Transformers (ViTs). This allows us to capitalize on their existingcapabilities and leverage the substantial cost of training them. We find thatjoint scaling of the vision and language components is important. Sinceexisting Transformers for language are much larger than their visioncounterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify thebenefits from even larger-capacity vision models. To train PaLI, we create alarge multilingual mix of pretraining tasks, based on a new image-text trainingset containing 10B images and texts in over 100 languages. PaLI achievesstate-of-the-art in multiple vision and language tasks (such as captioning,visual question-answering, scene-text understanding), while retaining a simple,modular, and scalable design.