Abstract
We present an efficient encoder-free approach for video-languageunderstanding that achieves competitive performance while significantlyreducing computational overhead. Current video-language models typically relyon heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4Bparameters), creating a substantial computational burden when processingmulti-frame videos. Our method introduces a novel Spatio-Temporal AlignmentBlock (STAB) that directly processes video inputs without requiring pre-trainedencoders while using only 45M parameters for visual processing - at least a6.5$\times$ reduction compared to traditional approaches. The STAB architecturecombines Local Spatio-Temporal Encoding for fine-grained feature extraction,efficient spatial downsampling through learned attention and separatemechanisms for modeling frame-level and video-level relationships. Our modelachieves comparable or superior performance to encoder-based approaches foropen-ended video question answering on standard benchmarks. The fine-grainedvideo question-answering evaluation demonstrates our model's effectiveness,outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in keyaspects like correctness and temporal understanding. Extensive ablation studiesvalidate our architectural choices and demonstrate the effectiveness of ourspatio-temporal modeling approach while achieving 3-4$\times$ faster processingspeeds than previous methods. Code is available at\url{https://github.com/jh-yi/Video-Panda}.