UniVLA: Learning to Act Anywhere with Task-centric Latent Actions
从视频中无监督学习任务中心潜在动作，实现跨体态通用机器人策略Unsupervised learning of task-centric latent actions from videos for cross-embodiment generalist robot policies
Qingwen Bu · Yanting Yang · Jisong Cai · Shenyuan Gao · Guanghui Ren · Maoqing Yao · Ping Luo · Hongyang Li
UniVLA 提出一种统一的视觉-语言-动作（VLA）框架：通过 inverse dynamics model 从视频帧对中以无监督方式提取任务中心潜在动作，使机器人策略能够直接利用无动作标注的多体态视频数据预训练，以不足 OpenVLA 1/20 的预训练计算量和 1/10 的下游数据量，在操作与导航任务上均超越现有方法。UniVLA presents a unified vision-language-action (VLA) framework: an inverse dynamics model extracts task-centric latent actions from video frame pairs in an unsupervised manner, letting robot policies be pretrained directly on action-free multi-embodiment video data. With less than 1/20 of OpenVLA's pretraining compute and 1/10 of its downstream data, it surpasses existing methods on both manipulation and navigation tasks.
现有 VLA 方法严重依赖带动作标注的数据集进行扩展，导致跨体态知识迁移困难、预训练成本高昂。如何像大语言模型学习跨语言共享知识一样，让机器人从无标注视频中学习通用动作表示？Existing VLA methods rely heavily on scaling action-annotated datasets, which makes cross-embodiment knowledge transfer difficult and pretraining expensive. How can robots learn generic action representations from unlabeled videos, the way large language models learn cross-lingual shared knowledge?
"most existing approaches heavily rely on scaling action-annotated data to enhance their capabilities … We draw inspiration from the fact that large language models learn cross-lingual shared knowledge … and propose UniVLA to derive task-centric action representations from videos."
图1：UniVLA 是一个统一的 VLA 框架，通过无监督方式从不同体态和视角的视频中提取任务中心潜在动作，无需动作标签即可利用任意体态的数据进行预训练。Figure 1: UniVLA is a unified VLA framework that extracts task-centric latent actions in an unsupervised manner from videos of diverse embodiments and viewpoints, enabling pretraining on data from any embodiment without action labels.
95.2%LIBERO 平均成功率 （Full 预训练）LIBERO average success rate (Full pretraining)
1/20相比 OpenVLA 所需 预训练计算量pretraining compute relative to OpenVLA
02方法 MethodMethod
UniVLA 分三个阶段：① 从视频帧对中学习任务中心潜在动作；② 预训练通用 VLA 策略预测潜在动作 token；③ 用轻量 action decoder 将潜在动作解码为各体态的真实机器人控制量。UniVLA proceeds in three stages: ① learn task-centric latent actions from video frame pairs; ② pretrain a generalist VLA policy that predicts latent action tokens; ③ decode latent actions into real robot control commands for each embodiment with a lightweight action decoder.
以成对视频帧为输入，通过 inverse dynamics model 配合 VQ-VAE 量化，将视觉变化压缩为离散潜在动作 token。关键创新在于利用 DINOv2 特征空间（而非原始像素）进行建模，天然过滤无关视觉变化（非自我运动体、不可预测的摄像头抖动等）。训练分两步进行：Stage 1 用语言指令作为 encoder 与 decoder 的条件输入，解耦任务相关动态；Stage 2 引入"任务无关"分支，强制 encoder 仅捕获任务相关的视觉变化。Taking paired video frames as input, an inverse dynamics model combined with VQ-VAE quantization compresses visual change into discrete latent action tokens. The key innovation is modeling in the DINOv2 feature space rather than in raw pixels, which naturally filters out irrelevant visual change (non-ego-motion agents, unpredictable camera shake, and the like). Training proceeds in two steps: Stage 1 conditions both encoder and decoder on the language instruction to disentangle task-relevant dynamics; Stage 2 introduces a "task-irrelevant" branch that forces the encoder to capture only task-relevant visual change.
图2：任务中心潜在动作学习框架。Stage 1 以语言指令作为 encoder/decoder 条件，Stage 2 增加任务无关分支进行解耦，确保潜在动作仅反映与任务指令相关的视觉动态变化。Figure 2: The task-centric latent action learning framework. Stage 1 conditions the encoder/decoder on the language instruction; Stage 2 adds a task-irrelevant branch for disentanglement, ensuring that latent actions reflect only the visual dynamics related to the task instruction.图8：潜在动作语义一致性分析。不同数据源与体态（机器人 / 人手）被赋予相同潜在动作编码的图像对，表现出语义一致的动作类型，验证了跨体态潜在动作空间的共享语义。Figure 8: Semantic consistency analysis of latent actions. Image pairs from different data sources and embodiments (robot / human hand) that receive the same latent action code exhibit semantically consistent action types, verifying the shared semantics of the cross-embodiment latent action space.
策略骨干网络采用 Prismatic-7B VLM，以视觉 embedding 与任务指令 token 为输入，自回归预测离散化的潜在动作 token。这一设计使策略可直接利用所有有视频和语言的数据（含人类示教视频、OpenX 跨体态数据）进行大规模预训练，彻底打破对动作标注的依赖。The policy backbone is a Prismatic-7B VLM that takes visual embeddings and task instruction tokens as input and autoregressively predicts discretized latent action tokens. This design lets the policy be pretrained at scale on any data with video and language, including human demonstration videos and OpenX cross-embodiment data, thoroughly breaking the dependence on action annotations.
图3：通用策略架构。基于 Prismatic-7B VLM，以投影视觉 embedding 和任务指令 token 为输入，自回归预测潜在动作 token。历史潜在动作序列被编码为 action history token 注入当前决策，提升时序一致性。Figure 3: The generalist policy architecture. Built on the Prismatic-7B VLM, it takes projected visual embeddings and task instruction tokens as input and autoregressively predicts latent action tokens. The history of latent actions is encoded into action history tokens and injected into the current decision, improving temporal consistency.
部署阶段，为每个目标体态训练一个轻量 action decoder（仅 12.6M 参数），将策略输出的潜在动作 token 解码为该体态的实际控制指令。解码器以视觉观测与历史潜在动作为输入，可独立于主策略高效适配新体态，支持 10Hz 闭环推理（RTX 4090）。At deployment, a lightweight action decoder (only 12.6M parameters) is trained for each target embodiment, decoding the latent action tokens emitted by the policy into that embodiment's actual control commands. The decoder takes visual observations and past latent actions as input and can be adapted to a new embodiment efficiently and independently of the main policy, supporting 10Hz closed-loop inference (RTX 4090).
DINOv2 特征空间的优势Advantages of the DINOv2 feature space
DINOv2 的语义特征对这些干扰因素天然鲁棒DINOv2 semantic features are naturally robust to these nuisance factors
消融实验证明：在 DINOv2 空间建模比像素空间（Genie 方式）平均高 6.9%Ablations show that modeling in DINOv2 space beats pixel space (the Genie approach) by 6.9% on average
历史潜在动作的作用The role of past latent actions
将历史动作编码为 action history token 输入策略Past actions are encoded as action history tokens and fed to the policy
LIBERO-Long 任务：有 vs 无历史动作 → 92.0% vs 88.1%LIBERO-Long task: with vs. without action history → 92.0% vs 88.1%
R2R 导航任务：有 vs 无历史动作 → 47.1% vs 30.6%（+16.5%）R2R navigation task: with vs. without action history → 47.1% vs 30.6% (+16.5%)
03实验 ExperimentsExperiments
在三类任务上评估：桌面操作（LIBERO benchmark）、真实机器人部署（Piper 7-DoF 机械臂）和视觉语言导航（R2R VLN-CE）。基线包括 OpenVLA、LAPA、MaIL、MDT、Diffusion Policy、Octo 等主流方法。Evaluated on three families of tasks: tabletop manipulation (LIBERO benchmark), real-robot deployment (Piper 7-DoF arm), and vision-language navigation (R2R VLN-CE). Baselines include OpenVLA, LAPA, MaIL, MDT, Diffusion Policy, Octo and other mainstream methods.
UniVLA 相比 LAPA 平均成功率提升 36.7%，在语义推理要求最高的 Stack Tower of Hanoi 任务上尤为突出（LAPA 26.7% → UniVLA 86.7%）。UniVLA raises the average success rate over LAPA by 36.7%, and stands out most on Stack Tower of Hanoi, the task with the highest semantic reasoning demand (LAPA 26.7% → UniVLA 86.7%).
图7：泛化性评测设置与结果。在三种场景下评估：Lighting Variation（光照变化）、Visual Distractor（视觉干扰）、Novel Object（新颖物体）。UniVLA 平均得分 2.49，远超 LAPA（1.36）和 OpenVLA（0.98）。Figure 7: Generalization evaluation setup and results. Three settings are assessed: Lighting Variation, Visual Distractor, and Novel Object. UniVLA scores 2.49 on average, far ahead of LAPA (1.36) and OpenVLA (0.98).
图6：R2R VLN-CE 上的 Oracle Success Rate。UniVLA 仅用单帧 RGB 输入加历史潜在动作，达到 47.1%，与使用全历史观测的 NaVid 持平，超越 OpenVLA 29.6%、LLaVA-Nav 33.1%。Figure 6: Oracle Success Rate on R2R VLN-CE. Using only a single RGB frame plus the latent action history, UniVLA reaches 47.1%, on par with NaVid which consumes the full observation history, and ahead of OpenVLA at 29.6% and LLaVA-Nav at 33.1%.
消融实验Ablation studies
以下消融实验揭示了各设计选择的重要性（数据均来自论文 Table III–V）：The ablations below reveal the importance of each design choice (all numbers taken from Table III–V of the paper):
潜在动作类型Latent action type
Spatial
Object
Goal
Long
平均Average
Genie（全部变化）Genie (all changes)
89.8
92.8
77.2
69.6
82.3
任务无关分量Task-irrelevant component
68.0
90.4
67.2
0.2
56.5
任务中心（本文）Task-centric (ours)
91.2
94.2
90.2
79.4
88.7
图9：数据可扩展性分析。随着预训练语料库从 Bridge-V2（单源）扩展至加入 OpenX 跨体态数据与无标注人类示教视频，下游性能持续提升，验证了 UniVLA 跨体态数据利用的有效性。Figure 9: Data scalability analysis. As the pretraining corpus grows from Bridge-V2 alone to also include OpenX cross-embodiment data and unlabeled human demonstration videos, downstream performance keeps improving, confirming the effectiveness of UniVLA's use of cross-embodiment data.
04局限性 LimitationsLimitations
说明：以下局限性由论文作者在 Limitation 章节明确陈述。Note:The limitations below are stated explicitly by the authors in the paper's Limitation section.
潜在动作粒度固定，复杂体态支持有限Fixed latent action granularity, limited support for complex embodiments
论文指出："The fixed granularity of the latent action and the predefined codebook size may not be optimal for all tasks or embodiments."本文主要在单臂操作场景下验证；扩展到双臂或灵巧手系统需要更复杂的动作建模，尚未得到充分探索。The paper states: "The fixed granularity of the latent action and the predefined codebook size may not be optimal for all tasks or embodiments." This work is validated mainly on single-arm manipulation; extending it to bimanual or dexterous-hand systems calls for more complex action modeling and has not yet been sufficiently explored.
依赖细粒度语言指令Dependence on fine-grained language instructions
任务中心潜在动作的学习依赖语言指令作为解耦信号，要求指令描述短时程具体动作，而非高层目标。尽管作者指出方法可处理不同粒度的指令，但对极稀疏或极抽象指令的鲁棒性尚待验证。Learning task-centric latent actions depends on language instructions as the disentangling signal, requiring instructions that describe short-horizon concrete actions rather than high-level goals. Although the authors note that the method can handle instructions of varying granularity, its robustness to extremely sparse or abstract instructions remains to be verified.
World model 能力未充分利用World model capability left underexploited
潜在动作 decoder 实质上是一个 world model，但本文未将其用于规划树搜索或与奖励模型结合做强化学习，留有较大的探索空间。The latent action decoder is essentially a world model, yet this work does not use it for planning tree search or combine it with a reward model for reinforcement learning, leaving considerable room for exploration.
In-context learning 能力尚待开发In-context learning capability yet to be developed
作者提出未来可将人类示教视频编码为潜在动作序列，用于零样本技能迁移（in-context learning），但该能力在本文中尚未实现。The authors suggest that human demonstration videos could in future be encoded into latent action sequences for zero-shot skill transfer (in-context learning), but this capability is not realized in this work.
