Abstract
Multimodal Large Language Models (MLLMs) have recently demonstratedremarkable perceptual and reasoning abilities, typically comprising a VisionEncoder, an Adapter, and a Large Language Model (LLM). The adapter serves asthe critical bridge between the visual and language components. However,training adapters with image-level supervision often results in significantmisalignment, undermining the LLMs' capabilities and limiting the potential ofMultimodal LLMs. To address this, we introduce Supervised Embedding Alignment(SEA), a token-level alignment method that leverages vision-languagepre-trained models, such as CLIP, to align visual tokens with the LLM'sembedding space through contrastive learning. This approach ensures a morecoherent integration of visual and language representations, enhancing theperformance and interpretability of multimodal LLMs while preserving theirinherent capabilities. Extensive experiments show that SEA effectively improvesMLLMs, particularly for smaller models, without adding extra data or inferencecomputation. SEA also lays the groundwork for developing more general andadaptable solutions to enhance multimodal systems.