将视觉、语言与动作统一为离散 token，通过世界模型预训练实现高效跨模态机器人策略学习Unifying vision, language, and action into discrete tokens, enabling efficient cross-modal robot policy learning via world-model pre-training
Yuqi Wang · Xinghang Li · Wenxuan Wang · Junbo Zhang · Yingyan Li · Yuntao Chen · Xinlong Wang · Zhaoxiang Zhang
UniVLA 以纯自回归 Transformer 将视觉、语言与动作信号编码为统一的离散 token 序列，并设计了两阶段训练策略：先在 622K 视频上进行无动作监督的世界模型后训练，再对下游任务进行策略微调。在 LIBERO 基准上以 95.5% 的平均成功率超越 π₀-FAST 的 85.5%，在 CALVIN 及 SimplerEnv 上也均达到新的最优水平。UniVLA encodes vision, language, and action signals into a unified sequence of discrete tokens with a purely autoregressive Transformer, and adopts a two-stage training strategy: action-free world-model post-training on 622K videos, followed by policy fine-tuning on downstream tasks. It reaches a 95.5% average success rate on the LIBERO benchmark, surpassing π₀-FAST's 85.5%, and also sets new state-of-the-art results on CALVIN and SimplerEnv.
VLAvision-language-action机器人操作robot manipulation世界模型world model自回归建模autoregressive modeling离散tokendiscrete tokensCALVINLIBERO多模态大模型multimodal large model策略学习policy learning
01动机Motivation
现有 VLA 方法大多采用"后期融合"策略：视觉特征提取与动作生成分处独立模块，难以捕捉跨模态的时序依赖关系。与此同时，有监督的机器人动作数据规模远小于互联网视频数据，策略泛化能力因此受限。如何充分利用海量无标注视频数据并实现视觉-语言-动作的紧密协同，是当前机器人学习面临的核心挑战。Most existing VLA methods adopt a "late fusion" strategy: visual feature extraction and action generation reside in separate modules, making cross-modal temporal dependencies hard to capture. At the same time, supervised robot action data is far smaller in scale than internet video data, which limits policy generalization. How to fully exploit massive unlabeled video data while achieving tight vision-language-action synergy is a core challenge in current robot learning.
"vision, language, and action are jointly modeled within a unified representation space to facilitate tighter cross-modal integration."
图 1：UniVLA 架构总览。所有模态（语言、视觉、动作）均被转化为统一词汇表中的离散 token，经单一自回归 Transformer 联合建模，支持动作预测、视觉未来帧预测及空间定位等多种任务。Figure 1: Overview of the UniVLA architecture. All modalities (language, vision, action) are converted into discrete tokens from a unified vocabulary and jointly modeled by a single autoregressive Transformer, supporting action prediction, future visual frame prediction, spatial grounding, and other tasks.
4.63CALVIN ABCD→D 平均任务链长度（SOTA）CALVIN ABCD→D average task-chain length (SOTA)
69.8%SimplerEnv WidowX 成功率 vs SpatialVLA 42.7%SimplerEnv WidowX success rate vs SpatialVLA 42.7%
81.7%NAVSIM PDMS 仅用单目前视摄像头NAVSIM PDMS with a monocular front-view camera only
02方法Method
UniVLA 的核心思想是将所有模态统一为离散 token：视觉通过 VQ 编码器以 8× 空间压缩率离散化，动作通过 FAST 对连续序列施加离散余弦变换（DCT）后编码为 1024 个 token，语言沿用标准 BPE 词汇表。三种模态共享同一词汇表，由特殊分隔符（boi/eoi 标示图像起止，boa/eoa 标示动作起止）明确边界，统一输入 8.5B 参数的自回归 Transformer，训练目标为标准 next-token prediction（cross-entropy loss）。The core idea of UniVLA is to unify all modalities as discrete tokens: vision is discretized by a VQ encoder at an 8× spatial compression rate; actions are encoded into 1024 tokens by FAST, which applies a discrete cosine transform (DCT) to the continuous sequence; language keeps the standard BPE vocabulary. The three modalities share one vocabulary, with boundaries explicitly marked by special delimiters (boi/eoi for image start/end, boa/eoa for action start/end), and are fed together into an 8.5B-parameter autoregressive Transformer whose training objective is standard next-token prediction (cross-entropy loss).
图 2：两阶段训练框架。第一阶段（世界模型后训练）：以 Emu3 预训练权重初始化，在 622K 机器人及通用视频上进行无动作监督的视觉 token 预测；第二阶段（策略微调）：在目标任务数据上以双帧交错视觉-动作序列进行动作 token 监督微调。Figure 2: The two-stage training framework. Stage 1 (world-model post-training): initialized from Emu3 pre-trained weights, the model performs action-free visual token prediction on 622K robot and general videos. Stage 2 (policy fine-tuning): the model is fine-tuned with action-token supervision on target-task data using dual-frame interleaved vision-action sequences.
统一多模态 TokenizationUnified Multimodal Tokenization
视觉 token 采用与 Emu3 相同设计的 VQ 编码器，将图像以 8× 压缩率编码为离散 token。动作 token 采用 FAST tokenizer——对连续动作序列施加 DCT 变换后映射至 1024 个离散 token，替换语言词汇表末尾的 1024 个 ID，从而与语言 token 共享同一词汇表，实现真正统一的序列建模。特殊分隔符 boi/eoi（图像起止）和 boa/eoa（动作起止）明确标示模态边界。Visual tokens use a VQ encoder of the same design as Emu3, encoding images into discrete tokens at an 8× compression rate. Action tokens use the FAST tokenizer—a DCT transform is applied to the continuous action sequence and then mapped to 1024 discrete tokens, which replace the last 1024 IDs of the language vocabulary so that actions share one vocabulary with language, enabling truly unified sequence modeling. Special delimiters boi/eoi (image start/end) and boa/eoa (action start/end) explicitly mark modality boundaries.
后训练（World Model Post-training）：模型以 Emu3 第一阶段预训练权重初始化，在 622K 精心策划的视频（覆盖 RT-1、BridgeV2、DROID 等多个机器人数据集及 Something-Something-V2 等通用视频）上训练，监督信号仅作用于视觉 token，无需动作标注。序列形式为 Sv = {Lt¹, Lv¹, Lv², …, Lvt}，语言指令条件化视觉预测，引导模型学习环境因果动态。训练 30K 步，batch size 64，在 32 块 A100 GPU 上耗时 4–5 天。Post-training (World Model Post-training): the model is initialized from Emu3 stage-1 pre-trained weights and trained on 622K carefully curated videos (covering robot datasets such as RT-1, BridgeV2 and DROID, as well as general videos such as Something-Something-V2), with the supervision signal applied to visual tokens only, requiring no action annotations. The sequence takes the form Sv = {Lt¹, Lv¹, Lv², …, Lvt}, where language instructions condition visual prediction and guide the model to learn environmental causal dynamics. Training runs for 30K steps with batch size 64, taking 4–5 days on 32 A100 GPUs.
微调（Policy Fine-tuning）：以后训练权重为起点，采用双帧交错的视觉-动作序列，action chunk size 为 10，损失仅作用于动作 token。余弦退火学习率从 8×10⁻⁵ 开始，在各基准上分别训练 8K–20K 步。Fine-tuning (Policy Fine-tuning): starting from the post-training weights, dual-frame interleaved vision-action sequences are used with an action chunk size of 10, and the loss is applied to action tokens only. The cosine-annealed learning rate starts at 8×10⁻⁵, and each benchmark is trained for 8K–20K steps.
图 3：UniVLA 的多模态能力展示，包括动作预测、视觉未来帧预测以及空间定位，体现了统一 token 建模框架的通用性——同一模型无需模态专用分支即可处理多种任务类型。Figure 3: A showcase of UniVLA's multimodal capabilities, including action prediction, future visual frame prediction, and spatial grounding, illustrating the generality of the unified token modeling framework—one model handles multiple task types without modality-specific branches.
03实验Experiments
在 CALVIN、LIBERO、SimplerEnv 和 NAVSIM 四个基准上与当前最优方法对比，并通过消融实验定量验证各关键组件的贡献。真实机器人实验在 AgileX 双臂平台上进行（3 路 RGB 摄像头，128×128 分辨率，14 维动作向量，chunk size 20），共 8 类操作任务。UniVLA is compared with state-of-the-art methods on four benchmarks—CALVIN, LIBERO, SimplerEnv and NAVSIM—and ablation studies quantitatively verify the contribution of each key component. Real-robot experiments are conducted on an AgileX dual-arm platform (3 RGB cameras, 128×128 resolution, 14-dimensional action vectors, chunk size 20) over 8 categories of manipulation tasks.
LIBERO 基准LIBERO Benchmark
方法Method
Spatial
Object
Goal
Long
Average
UniVLA（本文）UniVLA (ours)
95.4%
98.8%
93.6%
94.0%
95.5%
π₀-FAST
96.4%
96.8%
88.6%
60.2%
85.5%
CoT-VLA
87.5%
91.6%
87.6%
69.0%
81.1%
SpatialVLA
88.2%
89.9%
78.6%
55.5%
78.1%
CALVIN 基准（平均任务链长度，满分 5）CALVIN Benchmark (average task-chain length, max 5)
消融：世界模型后训练的贡献Ablation: Contribution of World-Model Post-training
后训练对性能提升贡献最为显著：移除后，LIBERO 整体从 94.2% 降至 48.5%（−45.7 pp），LIBERO-Long 从 89.2% 降至 17.4%（−71.8 pp），CALVIN ABCD→D 从 4.61 降至 1.46（−3.15）。数据效率实验表明：仅使用 10% 训练数据时，UniVLA 在 CALVIN 上仍达 3.19，显著高于 RoboVLMs 的 2.52，体现出世界模型预训练带来的强泛化能力。Post-training contributes most to the performance gain: removing it drops overall LIBERO from 94.2% to 48.5% (−45.7 pp), LIBERO-Long from 89.2% to 17.4% (−71.8 pp), and CALVIN ABCD→D from 4.61 to 1.46 (−3.15). Data-efficiency experiments show that with only 10% of the training data UniVLA still reaches 3.19 on CALVIN, markedly above the 2.52 of RoboVLMs, reflecting the strong generalization brought by world-model pre-training.
图 4：真实世界任务示例，包括擦白板、桌面整理、汉堡组装和连接器插入，展示 UniVLA 在 AgileX 双臂机器人平台上的多样化操作能力。Figure 4: Examples of real-world tasks, including whiteboard wiping, tabletop tidying, hamburger assembly and connector insertion, demonstrating the diverse manipulation capabilities of UniVLA on the AgileX dual-arm robot platform.
04局限性Limitations
Note:以下前两条为论文作者明确陈述（stated by the authors）；第三条为从设计层面推断（inferred from the design）。The first two items below are explicitly stated by the authors; the third is inferred from the design.
后训练规模化探索受限于算力Exploration of post-training scalability is limited by compute
作者指出："Due to limited computational resources, our investigation into post-training scalability is still in its early stages." 现有实验仅在 32 块 A100 上进行，更大规模的后训练数据与参数量对性能的影响尚未充分研究，存在进一步提升空间。The authors note: "Due to limited computational resources, our investigation into post-training scalability is still in its early stages." Existing experiments run on only 32 A100 GPUs; the impact of larger post-training data and parameter counts on performance has not been thoroughly studied, leaving room for further improvement.
与强化学习的整合有待深入Integration with reinforcement learning remains to be explored
论文承认当前方法需要 "further research to fully integrate it with reinforcement learning paradigms, enabling more robust and adaptive policy learning"。现阶段仅依赖模仿学习，RL 闭环训练（探索、奖励设计等）尚未实现。The paper acknowledges that the current method needs "further research to fully integrate it with reinforcement learning paradigms, enabling more robust and adaptive policy learning". At this stage it relies on imitation learning alone; closed-loop RL training (exploration, reward design, etc.) has not been realized.
视觉分辨率低 & 大模型实时部署挑战（inferred）Low visual resolution & real-time deployment challenges of a large model (inferred)
真实机器人实验采用 128×128 分辨率，在细粒度操作（如连接器插入）中细节信息损失明显。此外，8.5B 参数的自回归 Transformer 在对延迟敏感的实时控制场景下部署存在挑战，论文未提供推理速度或模型量化方案的讨论。Real-robot experiments use 128×128 resolution, and detail is noticeably lost in fine-grained manipulation such as connector insertion. Moreover, an 8.5B-parameter autoregressive Transformer is challenging to deploy in latency-sensitive real-time control scenarios, and the paper provides no discussion of inference speed or model quantization schemes.
