LingBot-VA 是一个自回归扩散框架，将视频帧预测（visual dynamics）与策略执行（policy execution）在统一的因果序列中联合建模，通过 KV-cache 保留完整历史上下文，在长时域操作、样本效率和泛化能力三个维度上全面超越现有 VLA 方法。LingBot-VA is an autoregressive diffusion framework that jointly models video frame prediction (visual dynamics) and policy execution within a unified causal sequence. By preserving the complete historical context through a KV-cache, it comprehensively surpasses existing VLA methods along three dimensions: long-horizon manipulation, sample efficiency, and generalization.
world modelautoregressive diffusionVLArobot manipulationMixture-of-TransformersKV-cache因果视频建模causal video modeling机器人操作robot manipulation
01动机Motivation
现有 Vision-Language-Action（VLA）策略直接从当前观测映射到动作，缺乏对物理世界动态的显式建模，导致长时域任务中容易遗忘历史、分布外泛化脆弱。这篇工作的核心主张是：Existing Vision-Language-Action (VLA) policies map directly from the current observation to actions, lacking an explicit model of physical world dynamics; as a result they easily forget history in long-horizon tasks and generalize poorly out of distribution. The core claim of this work is:
"Video world modeling, alongside vision-language pre-training, establishes a fresh and independent foundation for robot learning."
传统 VLA 学习的是直接映射 π(action|observation)，而 LingBot-VA 将其分解为两个步骤：先通过视觉动态预测 p(ot+1|o≤t) 生成下一帧，再通过逆动力学 g(at|ot, ot+1) 从相邻帧对推断动作。这一分解有三个关键优势：（1）保持对物理因果关系的显式约束；（2）通过 KV-cache 保留无限长历史，克服基于 chunk 方法的"短期记忆"问题；（3）大量互联网视频数据可直接用于预训练视觉动态模块。Conventional VLA learns a direct mapping π(action|observation), whereas LingBot-VA decomposes it into two steps: first predict the visual dynamics p(ot+1|o≤t) to generate the next frame, then infer the action from the adjacent frame pair through inverse dynamics g(at|ot, ot+1). This decomposition has three key advantages: (1) it maintains an explicit constraint on physical causality; (2) the KV-cache preserves unbounded history, overcoming the "short-term memory" problem of chunk-based methods; (3) large amounts of internet video data can be used directly to pre-train the visual dynamics module.
图1：LingBot-VA 概览。系统包含四个维度：(1) 在多样化视频与机器人数据上预训练；(2) 在真实世界和仿真任务上的综合评估结果；(3) 视觉动态预测等多元能力；(4) 时间记忆和少样本适应等涌现特性。Figure 1: Overview of LingBot-VA. The system covers four dimensions: (1) pre-training on diverse video and robot data; (2) comprehensive evaluation results on real-world and simulated tasks; (3) multiple capabilities such as visual dynamics prediction; (4) emergent properties such as temporal memory and few-shot adaptation.
98.5%LIBERO 平均成功率（新 SOTA）LIBERO average success rate (new SOTA)
+8.2%RoboTwin Easy Horizon=3 vs 前任 SOTARoboTwin Easy Horizon=3 vs the previous SOTA
+15.6%仅 10 个演示时 Make Breakfast progress score 相对 π₀.₅ 提升Make Breakfast progress score gain over π₀.₅ with only 10 demonstrations
50个真实演示即可完成 6 类操作任务后训练real-world demonstrations suffice to post-train on 6 categories of manipulation tasks
02方法Method
LingBot-VA 将视频 token 与动作 token 交错排列成统一因果序列，由一个 Mixture-of-Transformers（MoT）架构以自回归方式联合预测，同时配备闭环滚动推理机制和异步推理流水线以实现实时执行。LingBot-VA interleaves video tokens and action tokens into a unified causal sequence that a Mixture-of-Transformers (MoT) architecture predicts jointly in an autoregressive manner, equipped with a closed-loop rollout inference mechanism and an asynchronous inference pipeline for real-time execution.
图2：框架总览。LingBot-VA 采用自回归扩散，通过双流 Mixture-of-Transformers 架构（视频流：3072-dim；动作流：768-dim）实现统一的视频-动作世界建模。视频流预测未来的潜在视觉状态，动作流解码对应动作，两路通过 cross-modal fusion 耦合。Figure 2: Framework overview. LingBot-VA adopts autoregressive diffusion and achieves unified video-action world modeling through a dual-stream Mixture-of-Transformers architecture (video stream: 3072-dim; action stream: 768-dim). The video stream predicts future latent visual states and the action stream decodes the corresponding actions, the two branches being coupled by cross-modal fusion.
统一编码与 Mixture-of-Transformers 架构Unified encoding and the Mixture-of-Transformers architecture
视频帧通过因果 VAE 以 4×16×16 的时空压缩比编码为每帧 192 个 spatial token；动作通过轻量 MLP 映射到 token embedding，与视频 token 共享同一潜在空间。MoT 采用非对称容量设计：视频流 3072-dim，动作流 768-dim，两路通过 cross-modal attention 融合。统一的损失函数为：L = Ldyn + λ·Linv，其中 Ldyn 监督视频 token 的速度场（flow matching），Linv 以当前帧与下一帧为条件监督动作解码。Video frames are encoded by a causal VAE at a 4×16×16 spatiotemporal compression ratio into 192 spatial tokens per frame; actions are mapped by a lightweight MLP into token embeddings sharing the same latent space as the video tokens. MoT adopts an asymmetric capacity design: a 3072-dim video stream and a 768-dim action stream, fused through cross-modal attention. The unified loss is L = Ldyn + λ·Linv, where Ldyn supervises the velocity field of the video tokens (flow matching) and Linv supervises action decoding conditioned on the current and next frames.
因果注意力掩码与 Teacher Forcing 预训练Causal attention mask and Teacher Forcing pre-training
图3：Teacher Forcing Attention Mask。因果注意力掩码确保每个 token 只能 attend 到时序上在其之前的 token，从而在预训练中强制执行"物理动态的时间箭头"，同时允许对完整轨迹并行处理。Figure 3: Teacher Forcing Attention Mask. The causal attention mask ensures that each token can only attend to tokens preceding it in time, thereby enforcing the "arrow of time of physical dynamics" during pre-training while still allowing complete trajectories to be processed in parallel.
LingBot-VA 使用 Teacher Forcing 训练策略："Each token can only attend to preceding tokens in temporal sequence"，在强制因果一致性的同时实现高效的并行训练。训练时随机采样 chunk size K ∈ [1,8]，使得部署时可灵活权衡闭环修正频率与计算效率。另一关键技巧是 Noisy History Augmentation："During training, randomly augment video history with noise; enables partial denoising (s=0.5 instead of s=1.0) at inference, halving video generation steps."动作网络权重初始化采用视频权重插值并以 α=√(dv/da) 缩放，确保训练梯度平稳收敛。LingBot-VA uses a Teacher Forcing training strategy: "Each token can only attend to preceding tokens in temporal sequence", enforcing causal consistency while achieving efficient parallel training. During training the chunk size is randomly sampled as K ∈ [1,8], so that deployment can flexibly trade off closed-loop correction frequency against computational efficiency. Another key trick is Noisy History Augmentation: "During training, randomly augment video history with noise; enables partial denoising (s=0.5 instead of s=1.0) at inference, halving video generation steps." The action network weights are initialized by interpolating the video weights and rescaling by α=√(dv/da), ensuring smooth convergence of the training gradients.
异步推理流水线与 FDM GroundingAsynchronous inference pipeline and FDM grounding
图4：异步流水线设计。同步流水线因视频生成耗时导致动作执行延迟；Naive async 则因"open-loop degradation"使模型偏好时序平滑的幻想序列；FDM-grounded async 引入 Forward Dynamics Model 将真实观测重新对齐后再做预测，在速度与精度间取得平衡。Figure 4: Asynchronous pipeline design. The synchronous pipeline delays action execution because video generation is slow; naive async instead suffers from "open-loop degradation", making the model favour temporally smooth hallucinated sequences; FDM-grounded async introduces a Forward Dynamics Model that re-aligns real observations before predicting further, striking a balance between speed and accuracy.
部署阶段 KV-cache 将历史计算缓存，"Only new tokens require full attention computation; cached history tokens are reused"，显著降低推理延迟。为解决 Naive async 带来的开环退化问题，引入 Forward Dynamics Model（FDM）：模型利用最近的真实反馈想象施加动作后的视觉状态，强迫与环境观测重对齐后再向前预测。At deployment the KV-cache stores the history of computation — "Only new tokens require full attention computation; cached history tokens are reused" — markedly lowering inference latency. To solve the open-loop degradation introduced by naive async, a Forward Dynamics Model (FDM) is introduced: the model uses the most recent real feedback to imagine the visual state after the action is applied, forcing re-alignment with the environment observation before predicting forward.
注意：随 Horizon 增大，LingBot-VA 的优势进一步扩大（Horizon=3 时领先 +8.2% / +9.1%），印证了 KV-cache 长时记忆在多步任务中的核心价值。Note: as the Horizon grows, the advantage of LingBot-VA widens further (leading by +8.2% / +9.1% at Horizon=3), confirming the central value of KV-cache long-term memory in multi-step tasks.
LIBERO 基准评估LIBERO benchmark evaluation
方法Method
LIBERO-Spatial
LIBERO-Object
LIBERO-Goal
LIBERO-Long
平均Average
Octo
78.9
85.7
84.6
51.1
75.1
SmolVLA
93.0
94.0
91.0
77.0
88.8
π₀
96.8
98.8
95.8
85.2
94.1
X-VLA
98.2
98.6
97.8
97.6
98.1
LingBot-VA (Ours)
98.5±0.3
99.6±0.3
97.2±0.2
98.5±0.5
98.5
真实世界任务Real-world tasks
图5：真实世界部署结果。六类操作任务：Make Breakfast、Pick Screws（长时域）；Insert Tubes、Unpack Delivery（精细操作）；Fold Clothes、Fold Pants（可变形物体）。LingBot-VA 在成功率和 progress score 两个指标上均"substantially outperforming π₀.₅"。Figure 5: Real-world deployment results. Six categories of manipulation tasks: Make Breakfast and Pick Screws (long-horizon); Insert Tubes and Unpack Delivery (fine-grained manipulation); Fold Clothes and Fold Pants (deformable objects). On both success rate and progress score, LingBot-VA is "substantially outperforming π₀.₅".
样本效率Sample efficiency
图8：样本效率对比。仅使用 10 个演示时，LingBot-VA 在 Make Breakfast 任务上的 progress score 比 π₀.₅ 高 15.6%，在 RoboTwin 上高 10.3%。在多种数据规模下均保持优势。Figure 8: Sample efficiency comparison. With only 10 demonstrations, the progress score of LingBot-VA on the Make Breakfast task is 15.6% higher than π₀.₅, and 10.3% higher on RoboTwin. The advantage is maintained across a range of data scales.
时间记忆任务Temporal memory tasks
图9：时间记忆评估。Wipe Plate（需计数重复 6 次）和 Search Box（需记住空容器位置）两个任务。LingBot-VA 依靠完整 KV-cache 保留全部历史，"substantially outperforms π₀.₅ on both memory tasks"。Figure 9: Temporal memory evaluation. Two tasks: Wipe Plate (requires counting 6 repetitions) and Search Box (requires remembering the position of the empty container). Relying on the complete KV-cache to retain the entire history, LingBot-VA "substantially outperforms π₀.₅ on both memory tasks".
消融实验Ablation studies
消融维度Ablation dimension
设置Setting
Easy_all
Horizon=1
Horizon=2
Horizon=3
基准Baseline
LingBot-VA (Ours)
92.9
94.2
90.4
93.2
部署策略Deployment strategy
FDM-grounded Async
90.4
92.5
87.7
85.6
Naive Async
74.3
83.3
70.3
32.9
预训练主干Pre-training backbone
WAN (无因果世界建模)WAN (no causal world modeling)
80.6
84.9
76.3
67.6
消融结果清晰表明：Naive async 在 Horizon=3 时性能崩溃至 32.9%，而 FDM grounding 将其恢复至 85.6%；替换为普通 WAN 主干则损失 12.3 个百分点，证明因果世界建模预训练是核心贡献。The ablations show clearly that naive async collapses to 32.9% at Horizon=3, whereas FDM grounding restores it to 85.6%; replacing the backbone with plain WAN costs 12.3 percentage points, proving that causal world-modeling pre-training is the core contribution.
04局限性Limitations
说明：以下局限性部分为论文作者明确指出（标注"stated"），部分为从设计推断（标注"inferred"）。Note: Some of the limitations below are explicitly pointed out by the authors (marked "stated"), others are inferred from the design (marked "inferred").
视频 token 生成计算开销大 （stated）Video token generation is computationally expensive (stated)
论文明确指出："Video token generation remains computationally intensive"。自回归逐帧生成在部署时仍有较高延迟，即使通过异步流水线和 partial denoising 已显著缓解，实时性仍弱于直接动作回归方法。The paper states explicitly: "Video token generation remains computationally intensive". Frame-by-frame autoregressive generation still incurs high latency at deployment; even though the asynchronous pipeline and partial denoising alleviate this substantially, real-time performance remains weaker than that of direct action-regression methods.
接触动力学为隐式建模 （stated）Contact dynamics are modeled implicitly (stated)
论文明确指出当前版本为"implicit rather than explicit contact dynamics modeling"，对需要精确力/力矩控制的精细操作（如拧螺丝）可能存在上限。The paper explicitly notes that the current version performs "implicit rather than explicit contact dynamics modeling", which may impose a ceiling on fine-grained manipulation requiring precise force/torque control (such as screwing in a screw).
（推断）LingBot-VA 将动作统一为 30 维向量（双臂：7 EEF + 7 关节 + 1 夹爪，每臂），依赖于机器人形态一致性假设。对于腿式机器人、多指手等不同形态，需重新定义动作表示，框架的跨形态泛化能力尚未验证。(Inferred) LingBot-VA unifies actions into a 30-dimensional vector (bimanual: 7 EEF + 7 joints + 1 gripper per arm), which relies on an assumption of consistent robot embodiment. Different morphologies such as legged robots or multi-fingered hands require redefining the action representation, and the cross-embodiment generalization of the framework has not yet been validated.
论文将"multi-modal sensory inputs (tactile, force, audio) for robust manipulation"列为未来工作方向，当前系统仅依赖视觉输入，对无视觉线索的操作任务（如在遮挡下搜索）能力有限。The paper lists "multi-modal sensory inputs (tactile, force, audio) for robust manipulation" as a direction for future work. The current system relies on visual input alone, and its capability is limited on manipulation tasks without visual cues (such as searching under occlusion).
