Abstract
In recent years, multimodal large language models (MLLMs) such as GPT-4V havedemonstrated remarkable advancements, excelling in a variety of vision-languagetasks. Despite their prowess, the closed-source nature and computationaldemands of such models limit their accessibility and applicability. This studyintroduces TinyGPT-V, a novel open-source MLLM, designed for efficient trainingand inference across various vision-language tasks, including image captioning(IC) and visual question answering (VQA). Leveraging a compact yet powerfularchitecture, TinyGPT-V integrates the Phi-2 language model with pre-trainedvision encoders, utilizing a unique mapping module for visual and linguisticinformation fusion. With a training regimen optimized for small backbones andemploying a diverse dataset amalgam, TinyGPT-V requires significantly lowercomputational resources 24GB for training and as little as 8GB for inferencewithout compromising on performance. Our experiments demonstrate thatTinyGPT-V, with its language model 2.8 billion parameters, achieves comparableresults in VQA and image inference tasks to its larger counterparts while beinguniquely suited for deployment on resource-constrained devices throughinnovative quantization techniques. This work not only paves the way for moreaccessible and efficient MLLMs but also underscores the potential of smaller,optimized models in bridging the gap between high performance and computationalefficiency in real-world applications. Additionally, this paper introduces anew approach to multimodal large language models using smaller backbones. Ourcode and training weights are available in the supplementary material.