Unified Video Action Model

01 动机Motivation

机器人学习领域长期面临一个根本矛盾：动作预测需要高时间频率的密集、精细推理，而视频生成需要高空间分辨率和大量计算资源。现有方法要么完全跳过视频生成（action-only policies），丢失了视觉场景动态的监督信号；要么先生成视频再预测动作（如 UniPi），导致推理速度慢且错误传播严重。Robot learning has long faced a fundamental tension: action prediction requires dense, fine-grained reasoning at high temporal frequency, whereas video generation requires high spatial resolution and large amounts of computation. Existing methods either skip video generation entirely (action-only policies), losing the supervisory signal of visual scene dynamics, or generate video first and then predict actions (e.g., UniPi), which makes inference slow and lets errors propagate severely.

"We propose a joint video-action latent representation and decouple video-action decoding to achieve both accuracy and computational efficiency."

UVA Teaser — **图1：Unified Video Action Model (UVA) 概览。** (a) UVA 采用联合视频-动作潜在表征与解耦解码架构：训练时视频和动作的 loss 同时监督共享 latent；部署时动作头独立运行，无需生成视频帧，实现高速推理。 (b) 通过 masked training，单一模型支持五种灵活的输入-输出组合，涵盖策略学习、视频生成、正/逆向动力学建模。**Figure 1: Overview of the Unified Video Action Model (UVA).** (a) UVA adopts a joint video-action latent representation with a decoupled decoding architecture: during training the video and action losses jointly supervise the shared latent; at deployment the action head runs on its own, requiring no video frame generation and achieving fast inference. (b) Through masked training, a single model supports five flexible input-output combinations, covering policy learning, video generation, and forward/inverse dynamics modeling.

+20%PushT Multitask 成功率提升（vs. 最优基线）PushT Multitask success-rate gain (vs. best baseline)

+5%Libero10 成功率提升（vs. π₀）Libero10 success-rate gain (vs. π₀)

95ms真实环境推理延迟（无需视频生成）Real-world inference latency (no video generation needed)

0.5B参数量（vs. π₀ 的 3.3B）Parameter count (vs. 3.3B for π₀)

02 方法Method

UVA 由四个核心模块组成：历史编码器（Encode History）、masked autoencoder 观测预测器、解耦的视频与动作扩散解码头，以及支持多任务的 masked training 机制。训练时两种解码 loss 联合优化共享 latent；推理时只需运行轻量动作头，彻底消除视频生成的推理开销。UVA consists of four core modules: a history encoder (Encode History), a masked autoencoder observation predictor, decoupled video and action diffusion decoding heads, and a masked training mechanism that supports multiple tasks. During training the two decoding losses jointly optimize the shared latent; at inference only the lightweight action head needs to be run, completely eliminating the inference overhead of video generation.

UVA Network Architecture — **图2：UVA 网络架构。** 历史观测帧经预训练 VAE（kl-f16）编码为 latent map，展平后投影为 N 维视觉 token；动作块对齐重复后同样投影为 N 个动作 token。未来帧经随机 masking 后送入 Transformer，得到联合视频-动作潜在表征 {Z}。随后两个独立扩散头分别重建视频帧和生成动作序列。语言条件任务（如 Libero10）通过 CLIP 编码后追加到 token 序列。**Figure 2: UVA network architecture.** History observation frames are encoded by a pretrained VAE (kl-f16) into a latent map, which is flattened and projected into N-dimensional visual tokens; action chunks are repeated for alignment and likewise projected into N action tokens. Future frames are randomly masked and fed into the Transformer, yielding the joint video-action latent representation {Z}. Two independent diffusion heads then reconstruct video frames and generate action sequences respectively. For language-conditioned tasks (e.g., Libero10), the language is CLIP-encoded and appended to the token sequence.

III-A 历史编码 (Encode History)III-A Encode History

历史观测帧 {O_t−h+1, …, O_t} 通过预训练 VAE 编码器（kl-f16）编码为形如 ℝ^w×h×c 的 latent map，展平并经全连接层投影为 d 维向量，每帧表示为 N 个视觉 token。对于动作（采样频率高于观测），将动作块重复 M 次与视觉 token 对齐，经 FC 层投影为 N 个动作 token，作为 Transformer 的条件输入。History observation frames {O_t−h+1, …, O_t} are encoded by a pretrained VAE encoder (kl-f16) into a latent map of the form ℝ^w×h×c, then flattened and projected by a fully connected layer into d-dimensional vectors, so that each frame is represented by N visual tokens. For actions (sampled at a higher frequency than observations), action chunks are repeated M times to align with the visual tokens and projected by an FC layer into N action tokens, which serve as conditioning input to the Transformer.

III-B Masked Autoencoder 观测预测III-B Masked Autoencoder Observation Prediction

未来观测帧经与历史相同的方式编码，训练时对视觉 token 进行随机 masking，模型学习重建被遮盖的 token。Transformer 融合视频与动作信息，输出联合视频-动作潜在表征 Z。关键设计：跨所有视频帧在相同位置进行 masking，防止信息泄露。推理时模型从空序列出发自回归生成完整视频。对于语言条件任务，CLIP 编码的语言 token 追加到输入序列。Future observation frames are encoded in the same way as the history; during training the visual tokens are randomly masked and the model learns to reconstruct the masked tokens. The Transformer fuses video and action information and outputs the joint video-action latent representation Z. A key design: masking is applied at the same positions across all video frames, which prevents information leakage. At inference the model starts from an empty sequence and autoregressively generates the complete video. For language-conditioned tasks, CLIP-encoded language tokens are appended to the input sequence.

III-C 解耦视频与动作扩散 (Decoupled Diffusion)III-C Decoupled Video and Action Diffusion

与先生成视频再预测动作的层级式方案不同，UVA 使用两个独立的轻量级扩散解码器，均以共享 latent Z 为条件。训练时两路 loss 同步监督：Unlike hierarchical schemes that first generate video and then predict actions, UVA uses two independent lightweight diffusion decoders, both conditioned on the shared latent Z. During training the two losses supervise it simultaneously:

动作 loss：Action loss: L_action(Z,A) = E_ε,k[‖ε − ε_θ(A^(k)|k,Z)‖²]
视频 loss：Video loss: L_video(Z,O) = E_ε,k[(1/N)Σ‖ε_i − ε_φ(O^(i,k)|k,z_i)‖²]
总 loss：Total loss: L = L_action + L_video

部署时只运行动作扩散头，无需执行视频生成，推理延迟与纯动作策略相当。At deployment only the action diffusion head is run; no video generation is performed, and the inference latency is comparable to that of an action-only policy.

III-D Masked Training 多任务灵活目标III-D Masked Training with Flexible Multi-Task Objectives

通过在输入端对未使用的模态进行 masking 并替换为可学习的 mask token，单一模型支持五种训练任务的灵活切换：By masking the unused modalities at the input and replacing them with a learnable mask token, a single model supports flexible switching among five training tasks:

策略学习 (Policy Learning)Policy Learning

历史观测 + 历史动作 → 预测未来动作。核心机器人控制任务。History observations + history actions → predict future actions. The core robot control task.

视频生成 (Video Prediction)Video Prediction

历史观测 + 历史动作 → 预测未来视频帧。视觉场景规划与想象。History observations + history actions → predict future video frames. Visual scene planning and imagination.

正向动力学 (Forward Dynamics)Forward Dynamics

历史观测 + 历史动作 + 未来动作 → 预测未来视频。用于动作采样评分。History observations + history actions + future actions → predict future video. Used for scoring sampled actions.

逆向动力学 (Inverse Dynamics)Inverse Dynamics

历史观测 + 未来观测 → 预测连接动作。无需显式动作标注的场景。History observations + future observations → predict the connecting actions. For settings without explicit action labels.

03 实验Experiments

实验覆盖仿真与真实场景的单任务/多任务设置，评估 UVA 在策略学习、视频生成、正向/逆向动力学建模等方面的能力，并与 Diffusion Policy (DP-C, DP-UMI)、UniPi、π₀、OpenVLA 等基线进行对比。The experiments cover single-task and multi-task settings in both simulation and the real world, evaluating UVA on policy learning, video generation, and forward/inverse dynamics modeling, and comparing it against baselines such as Diffusion Policy (DP-C, DP-UMI), UniPi, π₀ and OpenVLA.

策略学习结果 (Policy Learning)Policy Learning Results

场景Setting	任务Task	最优基线Best baseline	UVA	备注Notes
仿真单任务Sim. single-task	PushT	0.91 (DP-C)	0.98	+7.7%
仿真单任务Sim. single-task	Toolhang	0.95 (DP-C)	0.88	略低于最优Slightly below the best
仿真多任务Sim. multi-task	PushT-M	0.68 (DP-C)	0.88	+20%
仿真多任务Sim. multi-task	Libero10	0.85 (π₀)	0.90	+5.9%，参数量仅 1/6+5.9%, only 1/6 the parameters
真实单任务Real-world single-task	UMI Cup	0.95 (DP-UMI)	0.85	DP-UMI 含恢复数据优势DP-UMI benefits from recovery data
真实多任务 (OOD)Real-world multi-task (OOD)	Cup	0.50 (DP-UMI)	0.65	分布外泛化Out-of-distribution generalization
真实多任务 (OOD)Real-world multi-task (OOD)	Mouse	0.40 (DP-UMI)	0.80	未见物体/夹爪Unseen objects/grippers

Real-World Out-of-Distribution Evaluation — **图3：真实场景分布外泛化评估。** 测试场景与训练数据存在显著分布偏移，包括未见过的环境、物体和机械夹爪。UVA 在 Cup（0.65 vs. 0.50）和 Mouse（0.80 vs. 0.40）任务上均明显超越 DP-UMI 基线，体现了联合视频-动作表征带来的更强泛化能力。**Figure 3: Real-world out-of-distribution generalization evaluation.** The test scenes exhibit a substantial distribution shift from the training data, including unseen environments, objects and mechanical grippers. UVA clearly surpasses the DP-UMI baseline on both Cup (0.65 vs. 0.50) and Mouse (0.80 vs. 0.40), reflecting the stronger generalization brought by the joint video-action representation.

视觉鲁棒性与历史长度鲁棒性Visual Robustness and History-Length Robustness

Visual Robustness on PushT — **图4：PushT 视觉干扰下的鲁棒性对比。** 测试条件包括背景颜色变化、分散注意力的背景物体、以及目标颜色变化。在目标颜色改变时，UVA 达到 64% 成功率，显著优于 UniPi（40%）和 OpenVLA（32%）。视频生成的辅助监督帮助模型学到对视觉干扰更鲁棒的表征。**Figure 4: Robustness comparison on PushT under visual disturbances.** The test conditions include background color changes, distracting background objects, and target color changes. When the target color changes, UVA reaches a 64% success rate, significantly outperforming UniPi (40%) and OpenVLA (32%). The auxiliary supervision from video generation helps the model learn representations that are more robust to visual disturbances.

在 PushT-M 的历史长度消融实验（history length 从 1 增加到 5）中，DP-C 随历史增加性能明显下降，而 UVA "maintained robust performance as history length increased"，展现出更稳定的时序建模能力。In the history-length ablation on PushT-M (history length increased from 1 to 5), DP-C degrades noticeably as the history grows, whereas UVA "maintained robust performance as history length increased", showing more stable temporal modeling.

视频生成质量 (FVD)Video Generation Quality (FVD)

场景Setting	UniPi FVD ↓	UVA FVD (1-step) ↓	UVA FVD (8-step) ↓
Libero10（仿真）Libero10 (simulation)	56.55	—	51.10
Cup Arrangement（真实）Cup Arrangement (real-world)	71.37	51.34	29.72

Video Generation Results — **图5：验证集视频生成质量对比（8 步自回归生成）。** UVA 生成的视频帧与 ground truth 高度吻合，时序一致性强。UniPi 偶尔生成模糊帧或物体缺失的帧。UVA 使用 8 步扩散在 Cup Arrangement 上达到 FVD 29.72（vs. UniPi 71.37）。**Figure 5: Video generation quality comparison on the validation set (8-step autoregressive generation).** The video frames generated by UVA agree closely with the ground truth and are temporally consistent. UniPi occasionally produces blurry frames or frames with missing objects. Using 8-step diffusion, UVA reaches FVD 29.72 on Cup Arrangement (vs. UniPi 71.37).

正向动力学模型 (Forward Dynamics)Forward Dynamics Model

在积木推拨任务（Block Pushing）中，UVA 的正向动力学模型为 DP-C 的 100 条采样动作轨迹打分选优。成功率从 DP-C 独立运行的 38% 提升至 60%（ground-truth 仿真器上限为 75%），四种颜色配置下平均提升 +22 个百分点。On the Block Pushing task, UVA's forward dynamics model scores and selects among 100 sampled action trajectories from DP-C. The success rate rises from 38% for DP-C alone to 60% (the ground-truth simulator upper bound is 75%), an average gain of +22 percentage points across the four color configurations.

逆向动力学模型 (Inverse Dynamics)Inverse Dynamics Model

方法Method	位置误差 (cm) ↓Position error (cm) ↓	旋转误差 (°) ↓Rotation error (°) ↓
UniPi 逆向动力学UniPi inverse dynamics	1.92	2.21
Visual-Inertial SLAM	0.41	0.30
UVA（本文）UVA (ours)	0.75	1.11

作者认为 UVA 的逆向动力学性能代表了 "a viable alternative to SLAM, which is difficult to calibrate and suffers from a high failure rate."The authors argue that UVA's inverse dynamics performance represents "a viable alternative to SLAM, which is difficult to calibrate and suffers from a high failure rate."

消融实验Ablation Studies

移除视频生成分支（UVA-action only）后，策略成功率在多任务设置下明显下降，验证了联合视频-动作监督对策略鲁棒性的贡献。对 masking 策略的消融（application-dependent vs. application-independent，不同 mask ratio）也在附录 Table VIII 中详细报告。在 Libero10 上加入少量人类示教视频（action-free），成功率从 0.90 进一步提升至 0.91（500-test 设置），证明框架具备利用无动作视频数据的潜力。Removing the video generation branch (UVA-action only) markedly lowers policy success rates in the multi-task setting, confirming the contribution of joint video-action supervision to policy robustness. Ablations of the masking strategy (application-dependent vs. application-independent, different mask ratios) are also reported in detail in appendix Table VIII. On Libero10, adding a small amount of human demonstration video (action-free) further raises the success rate from 0.90 to 0.91 (500-test setting), showing that the framework has the potential to exploit action-free video data.

推理速度Inference Speed

仿真任务单条轨迹推理耗时 0.23s（对比 DP-C Transformer 变体 0.36s）；真实世界实验推理延迟 95ms。"The use of decoupled diffusion heads eliminates the need for video generation during policy inference."Inference for a single trajectory on the simulation tasks takes 0.23s (compared with 0.36s for the DP-C Transformer variant); the real-world inference latency is 95ms. "The use of decoupled diffusion heads eliminates the need for video generation during policy inference."

04 局限性Limitations

说明： 以下局限性中，第一条为论文 Discussion 章节明确陈述（stated），其余为从方法设计中合理推断（inferred from design）。Note: Among the limitations below, the first is explicitly stated in the paper's Discussion section (stated); the rest are reasonably inferred from the method design (inferred from design).

无动作视频数据的利用尚不充分（stated）Action-free video data is not yet fully exploited (stated)

论文明确指出，当前框架 "does not currently leverage large amounts of actionless video data, which could provide valuable additional supervision." 作者建议通过在大规模网络视频数据集上进行预训练，可以显著增强模型的泛化能力。附录实验表明加入少量人类视频数据可小幅提升性能，但系统性探索留待未来工作。The paper explicitly notes that the current framework "does not currently leverage large amounts of actionless video data, which could provide valuable additional supervision." The authors suggest that pretraining on large-scale web video datasets could significantly strengthen the model's generalization. Appendix experiments show that adding a small amount of human video data yields a modest performance gain, but a systematic exploration is left to future work.

真实单任务性能略逊于专项基线（stated）Real-world single-task performance slightly trails specialized baselines (stated)

在真实环境 UMI Cup 单任务测试中，UVA 成功率 0.85 低于 DP-UMI 的 0.95。作者将此归因于 DP-UMI 使用了专为短历史窗口优化的恢复数据，而 UVA 使用多任务通用设置。此局限在多任务 OOD 场景中不再出现。In the real-world UMI Cup single-task test, UVA's success rate of 0.85 falls below DP-UMI's 0.95. The authors attribute this to DP-UMI's use of recovery data optimized specifically for short history windows, whereas UVA uses a general multi-task setting. This limitation no longer appears in the multi-task OOD setting.

模态扩展有限（inferred from design）Limited modality coverage (inferred from design)

当前 UVA 仅支持视觉观测与末端执行器动作两种模态。论文提到未来计划通过增加新的扩散头来扩展预测模态，"such as sound and force"，但当前版本尚不支持触觉、声音等多感官输入，限制了其在精密接触操作场景的应用。UVA currently supports only two modalities: visual observations and end-effector actions. The paper mentions plans to extend the predicted modalities by adding new diffusion heads, "such as sound and force", but the current version does not yet support multi-sensory inputs such as touch and sound, which limits its application to precise contact-rich manipulation.

视频生成与动作精度之间的权衡（inferred from design）Trade-off between video generation and action precision (inferred from design)

尽管解耦设计避免了推理时的视频生成开销，但训练时视频解码器与动作解码器共享 latent Z，两者优化目标存在潜在竞争。在部分高精度操作任务（如 Toolhang）上，UVA 成功率（0.88）略低于专项动作策略 DP-C（0.95），表明联合优化可能在极高精度场景下存在一定代价。Although the decoupled design avoids the cost of video generation at inference, during training the video decoder and the action decoder share the latent Z, so their optimization objectives potentially compete. On some high-precision manipulation tasks (e.g., Toolhang), UVA's success rate (0.88) is slightly below that of the specialized action policy DP-C (0.95), suggesting that joint optimization may carry a cost in extremely high-precision settings.