SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

  • 2023-11-13 18:59:47
  • Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, Yu Qiao
  • 0


We present SPHINX, a versatile multi-modal large language model (MLLM) with ajoint mixing of model weights, tuning tasks, and visual embeddings. First, forstronger vision-language alignment, we unfreeze the large language model (LLM)during pre-training, and introduce a weight mix strategy between LLMs trainedby real-world and synthetic data. By directly integrating the weights from twodomains, the mixed LLM can efficiently incorporate diverse semantics withfavorable robustness. Then, to enable multi-purpose capabilities, we mix avariety of tasks for joint visual instruction tuning, and design task-specificinstructions to avoid inter-task conflict. In addition to the basic visualquestion answering, we include more challenging tasks such as region-levelunderstanding, caption grounding, document layout detection, and human poseestimation, contributing to mutual enhancement over different scenarios.Additionally, we propose to extract comprehensive visual embeddings fromvarious network architectures, pre-training paradigms, and informationgranularity, providing language models with more robust image representations.Based on our proposed joint mixing, SPHINX exhibits superior multi-modalunderstanding capabilities on a wide range of applications. On top of this, wefurther propose an efficient strategy aiming to better capture fine-grainedappearances of high-resolution images. With a mixing of different scales andhigh-resolution sub-images, SPHINX attains exceptional visual parsing andreasoning performance on existing evaluation benchmarks. We hope our work maycast a light on the exploration of joint mixing in future MLLM research. Codeis released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.


Quick Read (beta)

loading the full paper ...