超越 VLA 的视频-动作模型通用机器人控制Video-Action Models for Generalizable Robot Control Beyond VLAs
Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, Elvis Nava
VLA（Vision-Language-Action）模型的预训练数据本质上是静态的，缺乏对物理动态的理解。
mimic-video 提出了一种新范式——Video-Action Model（VAM）：将大规模预训练视频模型（Cosmos-Predict2, 2B参数）
与轻量级 flow-matching 动作解码器配对，后者充当逆动力学模型（Inverse Dynamics Model），
在仿真和真实机器人操作任务上实现了最先进的性能，样本效率提升 10×，收敛速度提升 2×。The pretraining data of VLA (Vision-Language-Action) models is inherently static and lacks any understanding of physical dynamics.
mimic-video proposes a new paradigm, the Video-Action Model (VAM): a large-scale pretrained video model (Cosmos-Predict2, 2B parameters)
is paired with a lightweight flow-matching action decoder that acts as an inverse dynamics model (Inverse Dynamics Model),
achieving state-of-the-art performance on simulated and real robot manipulation tasks, with 10× better sample efficiency and 2× faster convergence.
当前主流 VLA 模型将视觉-语言预训练迁移到机器人控制，但其预训练数据（互联网图文）
本质上是静态的，无法捕捉物理世界的时序动态。这意味着机器人的物理理解能力必须完全从
稀缺且昂贵的机器人演示数据中从头学习。Current mainstream VLA models transfer vision-language pretraining to robot control, but their pretraining data (internet images and text)
is inherently static and cannot capture the temporal dynamics of the physical world. This means a robot's physical understanding must be learned entirely from scratch
from scarce and expensive robot demonstration data.
"the pretraining data, while massive in scale, is inherently static."
— 作者的核心出发点：用视频模型天然编码的时序动态替代静态视觉-语言预训练。"the pretraining data, while massive in scale, is inherently static."
— The authors' core starting point: replace static vision-language pretraining with the temporal dynamics that video models encode natively.
Oracle Case Study（图1）：
将动作解码器分别在预测的视频 latent和真实未来帧 latent两种条件下进行对比。
使用真实 latent 时，策略性能接近完美，表明策略性能直接与视频模型质量正相关——
视频模型越好，机器人控制性能的上限越高。Oracle Case Study (Fig. 1):
The action decoder is compared under two conditioning settings, predicted video latents and ground-truth future-frame latents.
With ground-truth latents the policy performs almost perfectly, showing that policy performance correlates directly with video model quality —
the better the video model, the higher the ceiling on robot control performance.
10×样本效率提升Sample efficiency gain vs. VLA baseline
2×收敛速度加速Convergence speedup vs. VLA baseline
56.3%SIMPLER-Bridge 平均成功率SIMPLER-Bridge average success rate （per-task τᵥ 设置）(per-task τᵥ setting)
mimic-video 将两个 Conditional Flow Matching（CFM）模型耦合：一个冻结的预训练视频骨干
（Cosmos-Predict2，经过 LoRA 微调）负责建模视觉时序动态；一个轻量级 DiT-based 动作解码器
从视频骨干提取的中间表示中预测机器人动作。两者在各自独立的 flow 时间调度（τᵥ 和 τₐ）下运行。mimic-video couples two Conditional Flow Matching (CFM) models: a frozen pretrained video backbone
(Cosmos-Predict2, LoRA-finetuned) models the visual temporal dynamics, while a lightweight DiT-based action decoder
predicts robot actions from the intermediate representations extracted from the video backbone. The two operate under separate flow time schedules (τᵥ and τₐ).
系统架构（图2）：
mimic-video 整体流程。视频骨干（Cosmos-Predict2）在视频 flow 时间 τᵥ 处执行部分去噪（partial denoising），
提取第 k 层的中间隐状态 htv。
动作解码器（DiT）以该隐状态为条件，通过 cross-attention 进行完整动作去噪（τₐ 从噪声到干净动作）。
两个 flow 调度器相互独立，允许在推理时灵活控制视频与动作的去噪程度。System architecture (Fig. 2):
The overall mimic-video pipeline. The video backbone (Cosmos-Predict2) performs partial denoising at video flow time τᵥ,
and the intermediate hidden state htv is extracted from layer k.
The action decoder (DiT) conditions on that hidden state and performs full action denoising through cross-attention (τₐ runs from noise to clean actions).
The two flow schedulers are mutually independent, allowing flexible control of how far video and actions are denoised at inference time.
核心创新：Partial Denoising 策略Key innovation: the Partial Denoising strategy
与直觉相反，作者发现最优性能出现在最高噪声水平（τᵥ ≈ 1），
而非完整视频重建（τᵥ = 0）。原因在于：Counterintuitively, the authors find that the best performance appears at the highest noise level (τᵥ ≈ 1),
rather than at full video reconstruction (τᵥ = 0). The reasons are:
分布对齐（Distribution Alignment）：
完整去噪后的视频预测会偏离训练分布，导致动作解码器收到 out-of-distribution 的条件信号。Distribution alignment:
Fully denoised video predictions drift away from the training distribution, so the action decoder receives out-of-distribution conditioning signals.
表示丰富性：
去噪流程末端的骨干层趋向近似恒等映射，中间隐状态反而包含更丰富的语义与动态信息。
Oracle 实验表明 τᵥ ≈ 0.4 时的 action reconstruction MSE 最低。Representation richness:
Backbone layers at the end of the denoising process tend toward a near-identity mapping, while the intermediate hidden states carry richer semantic and dynamic information.
Oracle experiments show that the action reconstruction MSE is lowest at τᵥ ≈ 0.4.
推理效率：
τᵥ = 1 时只需单次前向传播，同时实现最高性能和最快推理速度。Inference efficiency:
At τᵥ = 1 only a single forward pass is required, giving both the highest performance and the fastest inference.
Flow Matching 框架The Flow Matching framework
系统采用最优传输路径在干净数据与高斯噪声之间插值：
xᵗ = (1−τ)x⁰ + τε，τ ∈ [0,1]。
模型学习向量场 vθ 以回归条件生成场。
动作解码器为 DiT-based 架构，具备：
cross-attention（对接视频隐状态）、self-attention（跨动作序列）、
AdaLN 调制（融合 τᵥ 和 τₐ），以及本体感知状态编码（训练时使用 learned mask tokens）。The system interpolates between clean data and Gaussian noise along an optimal-transport path:
xᵗ = (1−τ)x⁰ + τε, τ ∈ [0,1].
The model learns a vector field vθ that regresses the conditional generating field.
The action decoder is a DiT-based architecture with
cross-attention (to the video hidden states), self-attention (across the action sequence),
AdaLN modulation (fusing τᵥ and τₐ), and proprioceptive state encoding (using learned mask tokens during training).
两阶段训练Two-stage training
Stage 1 · 视频预训练Stage 1 · Video pretraining
在机器人视频数据集上对 Cosmos-Predict2 进行 LoRA 微调，学习机器人操作场景的时序动态。骨干在 Stage 2 中冻结。Cosmos-Predict2 is LoRA-finetuned on robot video datasets to learn the temporal dynamics of robot manipulation scenes. The backbone is frozen in Stage 2.
Stage 2 · 动作解码器Stage 2 · Action decoder
从头训练轻量级 DiT 动作解码器，以冻结视频骨干的中间隐状态为条件，学习动作预测。
流程解耦使动作数据需求极低。A lightweight DiT action decoder is trained from scratch, conditioned on the intermediate hidden states of the frozen video backbone, to learn action prediction.
Decoupling the pipeline keeps the demand for action data extremely low.
03实验Experiments
评估覆盖三个设置：SIMPLER-Bridge 仿真基准（跨任务泛化）、LIBERO 仿真基准（语言条件操作）、
以及真实双臂灵巧操作（Franka 机械臂 + 16-DoF 灵巧手）。基线包括 OpenVLA、Octo、
FLOWER、Diffusion Policy、OpenVLA-OFT 及论文自行实现的 π₀.₅-style VLA。The evaluation covers three settings: the SIMPLER-Bridge simulation benchmark (cross-task generalization), the LIBERO simulation benchmark (language-conditioned manipulation),
and real-world bimanual dexterous manipulation (Franka arms + a 16-DoF dexterous hand). Baselines include OpenVLA, Octo,
FLOWER, Diffusion Policy, OpenVLA-OFT and the authors' own π₀.₅-style VLA implementation.
注：mimic-video 仅使用单一工作空间视角，且仅用 1h 33m（512 episodes）sorting 数据和 2h 14m（480 episodes）stowing 数据训练，
即超越使用腕部相机的多视角基线。Note: mimic-video uses only a single workspace view and is trained on just 1h 33m (512 episodes) of sorting data and 2h 14m (480 episodes) of stowing data,
yet already surpasses multi-view baselines that use wrist cameras.
真实双臂操作环境（图3）：
Franka 机械臂搭配 16-DoF 灵巧手执行包裹分拣（sorting）和胶带收纳（stowing）任务。
mimic-video 仅使用单一工作空间视角，无需腕部相机，即显著超越多视角基线。Real-world bimanual setup (Fig. 3):
Franka arms equipped with a 16-DoF dexterous hand perform package sorting and tape stowing tasks.
mimic-video uses only a single workspace view and no wrist cameras, yet clearly surpasses multi-view baselines.
样本效率与收敛速度Sample efficiency and convergence speed
样本效率（图4）：
mimic-video 仅需 10% 的动作训练数据即可达到 π₀.₅-style VLA baseline 的峰值性能。
仅用 2% 数据（每任务一条演示）时，平均成功率仍达 77%，与 Diffusion Policy baseline 持平。Sample efficiency (Fig. 4):
mimic-video needs only 10% of the action training data to match the peak performance of the π₀.₅-style VLA baseline.
With just 2% of the data (one demonstration per task), the average success rate still reaches 77%, on par with the Diffusion Policy baseline.收敛速度（图5）：
动作解码器在相同 batch size 和最优学习率下，收敛速度显著快于 VLA baseline，
且最终渐近成功率更高。Convergence speed (Fig. 5):
Under the same batch size and optimal learning rate, the action decoder converges markedly faster than the VLA baseline,
and reaches a higher final asymptotic success rate.
消融实验：视频 Flow 时间 τᵥ 的影响Ablation: the effect of the video flow time τᵥ
τᵥ 消融（图6）：
性能在中间噪声水平处达峰，τᵥ=1（单次前向传播）效果最佳。
Action reconstruction MSE 在 τᵥ ≈ 0.4 时最低（Oracle 实验）。
完全去噪（τᵥ=0）因分布偏移而导致性能下降。τᵥ ablation (Fig. 6):
Performance peaks at intermediate noise levels, and τᵥ=1 (a single forward pass) works best.
The action reconstruction MSE is lowest at τᵥ ≈ 0.4 (Oracle experiment).
Full denoising (τᵥ=0) degrades performance because of distribution shift.
其他关键消融发现：Other key ablation findings: • 最优视频源层： 第 k=19 层性能最强，"decreasing success rates towards initial or final layers"。• Optimal video source layer: layer k=19 performs best, with "decreasing success rates towards initial or final layers". • 观察帧数： 五帧历史优于单帧条件。• Number of observation frames: a five-frame history outperforms single-frame conditioning. • VLA baseline 层选择： FAST 预训练 VLM 的第 11 层 cross-attention 在 SIMPLER-Bridge 上表现最优。• VLA baseline layer choice: cross-attention at layer 11 of the FAST-pretrained VLM performs best on SIMPLER-Bridge.
04局限性Limitations
说明： 以下局限性均为作者在论文中明确陈述。Note: All limitations below are explicitly stated by the authors in the paper.
当前方法依赖"single-view video backbone, which restricts policies to a fixed, single workspace view"。
尽管这一限制使 mimic-video 在无需额外视角的情况下超越了多视角基线，
但也限制了其在需要动态视角或无固定工作空间环境中的适用性。The current method relies on a "single-view video backbone, which restricts policies to a fixed, single workspace view".
Although this restriction still lets mimic-video surpass multi-view baselines without extra viewpoints,
it also limits its applicability in settings that need dynamic viewpoints or that have no fixed workspace.
跨实体泛化尚未验证（Cross-Embodiment Scaling）Cross-Embodiment Scaling Not Yet Validated
作者明确指出尚未将 VAM 范式应用于统一的大规模跨实体模型：
"not yet applied the VAM recipe to train a unified, large-scale, cross-embodiment model"。
目前的实验仅在特定机械臂平台上验证。The authors explicitly note that they have not yet applied the VAM paradigm to a unified, large-scale cross-embodiment model:
"not yet applied the VAM recipe to train a unified, large-scale, cross-embodiment model".
The present experiments are validated only on specific robot arm platforms.
真实机器人实验集中于少量精心设计的任务："current real-world experiments are limited to a focused set of tasks"。
更广泛的任务多样性、非结构化场景及长时序操控的泛化性尚待验证。The real-robot experiments focus on a small set of carefully designed tasks: "current real-world experiments are limited to a focused set of tasks".
Generalization to broader task diversity, unstructured scenes and long-horizon manipulation remains to be verified.
