Abstract
Multimodal Large Language Models (MLLMs) suffer from high computational costsdue to their massive size and the large number of visual tokens. In this paper,we investigate layer-wise redundancy in MLLMs by introducing a novel metric,Layer Contribution (LC), which quantifies the impact of a layer'stransformations on visual and text tokens, respectively. The calculation of LCinvolves measuring the divergence in model output that results from removingthe layer's transformations on the specified tokens. Our pilot experimentreveals that many layers of MLLMs exhibit minimal contribution during theprocessing of visual tokens. Motivated by this observation, we propose ShortV,a training-free method that leverages LC to identify ineffective layers, andfreezes visual token updates in these layers. Experiments show that ShortV canfreeze visual token in approximately 60\% of the MLLM layers, therebydramatically reducing computational costs related to updating visual tokens.For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B whilemaintaining superior performance. The code will be publicly available athttps://github.com/icip-cas/ShortV