Abstract
We introduce VLM-Lens, a toolkit designed to enable systematic benchmarking,analysis, and interpretation of vision-language models (VLMs) by supporting theextraction of intermediate outputs from any layer during the forward pass ofopen-source VLMs. VLM-Lens provides a unified, YAML-configurable interface thatabstracts away model-specific complexities and supports user-friendly operationacross diverse VLMs. It currently supports 16 state-of-the-art base VLMs andtheir over 30 variants, and is extensible to accommodate new models withoutchanging the core logic. The toolkit integrates easily with various interpretability and analysismethods. We demonstrate its usage with two simple analytical experiments,revealing systematic differences in the hidden representations of VLMs acrosslayers and target concepts. VLM-Lens is released as an open-sourced project toaccelerate community efforts in understanding and improving VLMs.