StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding

01 Motivation

MLLM 在长视频理解上的两大瓶颈：一是视频帧编码后产生的视觉 token 数量常超出 LLM 的上下文长度；二是存储这些视觉 token 的 KV cache 并在解码时逐一 attend，带来巨大的显存和计算开销。虽然长上下文 LLM 的发展缓解了第一个问题，但显存/计算效率的问题在边缘设备等真实场景下依然突出。MLLMs face two major bottlenecks in long-video understanding: first, the number of visual tokens produced after encoding video frames often exceeds the context length of the LLM; second, storing the KV cache of these visual tokens and attending to them one by one during decoding brings enormous memory and compute overhead. Although the development of long-context LLMs has alleviated the first problem, the memory/compute efficiency problem remains prominent in real-world scenarios such as edge devices.

更棘手的是，现有的视觉压缩方法大多依赖两个假设：要么已知视频的完整长度（离线处理），要么已知用户即将提出的问题（query-aware 压缩）。论文指出：More awkwardly, most existing visual compression methods rely on two assumptions: either the full length of the video is known (offline processing), or the question the user is about to ask is known (query-aware compression). The paper points out:

"Existing visual compression methods require either encoding the entire visual context before compression or having access to the questions in advance, which is impractical for long video understanding and multi-turn conversational settings."

已有的流式方法也各有短板：ReKV（Di et al., 2025）把所有帧的 KV cache 全部存下来做 in-context retrieval，显存开销随视频变长而不可控；LiveVLM（Ning et al., 2025）虽然做了压缩，但采用 FIFO 策略直接丢弃最早的 KV，会导致视频早期信息被彻底遗忘。StreamMem 想要的是：训练无关（training-free）+ query-agnostic + 固定显存预算，三者同时满足。Existing streaming methods also have their own shortcomings: ReKV (Di et al., 2025) keeps the KV cache of all frames for in-context retrieval, so the memory cost grows uncontrollably as the video gets longer; LiveVLM (Ning et al., 2025) does perform compression, but its FIFO policy simply discards the earliest KV, which causes information from the early part of the video to be completely forgotten. What StreamMem wants is: training-free + query-agnostic + a fixed memory budget, all satisfied at the same time.

02 Method

StreamMem 在每个时间步接收一段新的视频片段，先做输入帧过滤去冗余，再编码为视觉 token 并生成 KV，然后把新 KV 与上一步保留下来的压缩 KV memory 合并，经过一个压缩模块（剪枝 + 合并）得到新的固定大小 KV cache，供下一时间步继续使用。At every time step StreamMem receives a new video segment, first performs input frame filtering to remove redundancy, then encodes it into visual tokens and produces their KV, next merges the new KV with the compressed KV memory retained from the previous step, and passes the result through a compression module (pruning + merging) to obtain a new fixed-size KV cache to be used at the next time step.

输入帧过滤（Input Frame Filtering）Input Frame Filtering

在送入 MLLM 前，先用视觉编码器算出每帧的 embedding，对相邻帧计算余弦相似度；如果相似度超过阈值 δ（实验中取 δ=0.95 效果最好），就认为两帧冗余，直接取平均合并。这一步能防止静态场景或高帧率视频中大量相似帧把 KV cache "撑满"重复信息，从而保留 memory 的多样性。Before feeding frames into the MLLM, the vision encoder first computes an embedding for each frame, and the cosine similarity between adjacent frames is computed; if the similarity exceeds the threshold δ (δ=0.95 works best in the experiments), the two frames are considered redundant and are directly averaged and merged. This step prevents the large number of similar frames in static scenes or high-frame-rate videos from "stuffing" the KV cache with duplicated information, thereby preserving the diversity of the memory.

KV Cache Memory：用聊天模板 token 做 query 的代理KV Cache Memory: chat template tokens as a proxy for the query

压缩的核心难点是：query-agnostic 场景下，压缩时刻并不知道用户会问什么问题。StreamMem 的解法是用系统的聊天模板 token（<|im_end|><|im_start|>assistant\n）作为一个"通用 query"的代理 Q，与视觉 token 的 key 做交叉注意力：The core difficulty of compression is that in a query-agnostic setting the model does not know what the user will ask at the moment of compression. StreamMem's solution is to use the system's chat template tokens (<|im_end|><|im_start|>assistant\n) as a proxy Q for a "generic query", and to cross-attend it with the keys of the visual tokens:

A_tⁱ = Softmax( Q(K_tⁱ)^⊤ / √d )

由于预训练数据中视频字幕（captioning）任务大量存在，这个模板 token 会隐式地促使模型去"描述视频内容"，即便没有显式问题，模型也倾向于关注真正有信息量的视觉区域。论文用消融实验证实：这个"chat template query"和一个人工写的通用问题（"What is happening in the video?"）效果相近，说明它确实在扮演通用 query 的角色。Because video captioning tasks are abundant in the pretraining data, this template token implicitly drives the model to "describe the video content", so that even without an explicit question the model still tends to attend to the genuinely informative visual regions. An ablation study in the paper confirms that this "chat template query" performs comparably to a hand-written generic question ("What is happening in the video?"), showing that it does play the role of a generic query.

频级 KV 合并（Frame-wise KV Merging）Frame-wise KV Merging

除了剪枝，StreamMem 还为每一帧计算一个基于归一化注意力分数的加权 prototype key/value（K̄, V̄ = Σ α·K, Σ α·V），插入到该帧被保留 token 序列的中间位置，同时保留原有的位置编号，从而实现精细 token + 全局帧摘要的双重记忆。Besides pruning, StreamMem also computes for each frame a weighted prototype key/value based on normalized attention scores (K̄, V̄ = Σ α·K, Σ α·V), inserted at the middle position of the retained token sequence of that frame while keeping the original position indices, thereby realizing a dual memory of fine-grained tokens plus a global frame summary.

位置编码：YaRN 上下文窗口扩展Positional encoding: YaRN context window extension

为解决 MLLM 在长视频上位置编码泛化差的问题，StreamMem 采用 YaRN（Peng et al., 2023）技术扩展视觉上下文窗口，而不是像 ReKV / LiveVLM / InfiniPot-V 那样简单地对保留 token 重新分配位置 ID（这会丢失原有的时空对齐信息）。消融显示 YaRN 缩放因子 λ 的选择对性能敏感，不同模型需要不同的 λ（LLaVA-OneVision 用 λ=8，Qwen2-VL 用 λ=2，Qwen2.5-VL 因默认上下文更长而 λ=1，即不缩放）。To solve the poor generalization of positional encoding of MLLMs on long videos, StreamMem adopts the YaRN technique (Peng et al., 2023) to extend the visual context window, instead of simply reassigning position IDs to the retained tokens as ReKV / LiveVLM / InfiniPot-V do (which loses the original spatio-temporal alignment information). The ablation shows that the choice of the YaRN scaling factor λ is sensitive for performance, and different models need different λ (LLaVA-OneVision uses λ=8, Qwen2-VL uses λ=2, and Qwen2.5-VL uses λ=1, i.e. no scaling, because its default context is longer).

03 Experiments

在三个离线长视频理解基准（MLVU、EgoSchema、VideoMME）和两个流式视频问答基准（RVS-Ego、RVS-Movie）上，使用 LLaVA-OneVision-7B、Qwen2-VL-7B、Qwen2.5-VL-3B 三个开源 MLLM 验证。所有实验可在单张 A100 GPU 上完成。Validation is performed on three offline long-video understanding benchmarks (MLVU, EgoSchema, VideoMME) and two streaming video question-answering benchmarks (RVS-Ego, RVS-Movie), using the three open-source MLLMs LLaVA-OneVision-7B, Qwen2-VL-7B and Qwen2.5-VL-3B. All experiments can be completed on a single A100 GPU.

离线长视频理解（Table 1，节选）Offline long-video understanding (Table 1, excerpt)

论文原话总结：Summarized in the paper's own words:

"we observe that StreamMem outperforms the baselines on all benchmarks except the 'long' subset of VideoMME for Qwen2-VL-7B."

不同 KV 预算下的对比（Table 3，Qwen2-VL-7B，MLVU）Comparison under different KV budgets (Table 3, Qwen2-VL-7B, MLVU)

在 24K token 预算（不到 Full KV 一半大小）下，StreamMem 的 All 指标（66.3）已经超过 Full KV（65.9）。Under a 24K token budget (less than half the size of Full KV), the All metric of StreamMem (66.3) already exceeds that of Full KV (65.9).

流式视频问答（Table 2，LLaVA-OneVision-7B）Streaming video question answering (Table 2, LLaVA-OneVision-7B)

StreamMem 在这两个基准上并未全面超过所有 baseline（例如 InfiniPot-V 在 RVS-Ego Acc 上略高），但与"卸载全部 KV cache"的 ReKV oracle 相比具有很强竞争力，同时显存占用远低于 ReKV。StreamMem does not surpass every baseline across the board on these two benchmarks (InfiniPot-V is slightly higher on RVS-Ego Acc, for example), but it is strongly competitive against the ReKV oracle that "offloads the entire KV cache", while its memory footprint is far lower than that of ReKV.

消融研究Ablation studies

Proxy query 类型（Table 4）：真实用户 query 明显优于 query-agnostic 方法（尤其在 multi-detail 任务上），说明在不知道问题的情况下保留所有细节确实更难；而"chat template query"与人工设计的"generic text query"表现相近（66.9 vs 66.7，All 指标），验证了 chat template token 隐式承担了通用 query 的作用。Type of proxy query (Table 4): a real user query is clearly better than query-agnostic methods (especially on multi-detail tasks), showing that retaining every detail without knowing the question is indeed harder; meanwhile the "chat template query" performs comparably to a hand-designed "generic text query" (66.9 vs 66.7 on the All metric), verifying that the chat template token implicitly takes on the role of a generic query.

KV 合并策略（Table 5）：任何形式的帧级合并都优于不合并（No Merging All=65.6）；StreamMem 的加权合并（66.9）优于 LiveVLM 式的简单平均合并（66.3）。KV merging strategy (Table 5): any form of frame-level merging is better than no merging (No Merging All=65.6); the weighted merging of StreamMem (66.9) is better than LiveVLM-style plain average merging (66.3).

输入帧过滤阈值（Table 6）：δ=0.95 是效果最好的"sweet spot"（All=66.9），过滤本身相比不过滤（65.4）稳定带来提升。Input frame filtering threshold (Table 6): δ=0.95 is the best-performing "sweet spot" (All=66.9), and filtering itself brings a stable improvement over no filtering (65.4).

04 Limitations