Abstract
Vision-language models (VLMs) could power real-time assistants and autonomousagents, but they face a critical challenge: understanding near-infinite videostreams without escalating latency and memory usage. Processing entire videoswith full attention leads to quadratic computational costs and poor performanceon long videos. Meanwhile, simple sliding window methods are also flawed, asthey either break coherence or suffer from high latency due to redundantrecomputation. In this paper, we introduce StreamingVLM, a model designed forreal-time, stable understanding of infinite visual input. Our approach is aunified framework that aligns training with streaming inference. Duringinference, we maintain a compact KV cache by reusing states of attention sinks,a short window of recent vision tokens, and a long window of recent texttokens. This streaming ability is instilled via a simple supervised fine-tuning(SFT) strategy that applies full attention on short, overlapped video chunks,which effectively mimics the inference-time attention pattern without trainingon prohibitively long contexts. For evaluation, we build Inf-Streams-Eval, anew benchmark with videos averaging over two hours that requires dense,per-second alignment between frames and text. On Inf-Streams-Eval, StreamingVLMachieves a 66.18% win rate against GPT-4O mini and maintains stable, real-timeperformance at up to 8 FPS on a single NVIDIA H100. Notably, our SFT strategyalso enhances general VQA abilities without any VQA-specific fine-tuning,improving performance on LongVideoBench by +4.30 and OVOBench Realtime by+5.96. Code is available at https://github.com/mit-han-lab/streaming-vlm.