WorldVLA: Towards Autoregressive Action World Model

01 动机Motivation

当前机器人学习中，动作模型（VLA）与世界模型被视为两个相互独立的范式：VLA 从图像和文本预测动作，世界模型从图像和动作预测未来帧。这两者的能力存在天然互补性，但此前没有工作将它们真正统一在同一框架中并验证相互增益。In current robot learning, the action model (VLA) and the world model are treated as two mutually independent paradigms: the VLA predicts actions from images and text, while the world model predicts future frames from images and actions. Their capabilities are naturally complementary, yet no prior work had truly unified them within one framework and verified the mutual gains.

"We integrate action model and world model into a unified framework, demonstrate that action and image generation mutually enhance each other, and propose an attention mask strategy that selectively masks prior actions during the generation of the current action."

**Figure 1**：三种模型范式对比。(a) 动作模型：从图像理解生成动作；(b) 世界模型：从图像与动作理解生成未来图像；(c) WorldVLA：本文提出的统一动作世界模型，同时具备图像与动作的理解与生成能力。**Figure 1**: Comparison of three model paradigms. (a) Action model: generates actions from image understanding; (b) World model: generates future images from image and action understanding; (c) WorldVLA: the unified action world model proposed in this paper, which both understands and generates images and actions.

81.8%WorldVLA 512×512
LIBERO 平均 SRWorldVLA 512×512
LIBERO average SR

76.5%OpenVLA 基线
LIBERO 平均 SROpenVLA baseline
LIBERO average SR

−10%FVD 改善
vs. 纯世界模型（50 帧）FVD improvement
vs. world model only (50 frames)

+4%抓取成功率提升
vs. 同骨干动作模型Grasping success rate gain
vs. same-backbone action model

核心问题：动作块生成中的误差累积Core Problem: Error Accumulation in Action Chunk Generation

在自回归生成多步动作块（action chunk）时，每一步动作都依赖前一步的输出。若前一步出现误差，后续动作会受到污染，导致整体性能显著下降。实验表明，在不加干预的情况下加入 action chunking 会导致成功率下降 10–50 个百分点。When multi-step action chunks are generated autoregressively, every action depends on the output of the previous step. If the previous step is erroneous, the subsequent actions are contaminated, causing a marked drop in overall performance. Experiments show that adding action chunking without any intervention lowers the success rate by 10–50 percentage points.

02 方法Method

WorldVLA 基于 Chameleon 架构构建，将图像、文本、动作三类 token 统一在单一序列中进行自回归建模。动作模型负责从视觉和语言条件预测动作序列，世界模型在此基础上额外预测下一帧图像，两者共享参数，以多任务方式联合训练。WorldVLA is built on the Chameleon architecture, unifying image, text, and action tokens into a single sequence for autoregressive modeling. The action model predicts action sequences conditioned on vision and language, and the world model additionally predicts the next image frame on top of that; the two share parameters and are jointly trained in a multi-task fashion.

**Figure 2**：WorldVLA 整体架构。动作模型以文本和历史图像帧为条件生成动作序列；世界模型在动作之后继续预测下一帧图像。两个分支共享同一个自回归 transformer，token 序列在同一上下文窗口内处理。**Figure 2**: Overall architecture of WorldVLA. The action model generates action sequences conditioned on text and history image frames; the world model then continues to predict the next image frame after the actions. The two branches share the same autoregressive transformer, and the token sequence is processed within one context window.

Tokenizer 设计Tokenizer Design

图像 tokenizer：VQ-GAN，codebook 大小 8192，压缩比 16×Image tokenizer: VQ-GAN, codebook size 8192, compression ratio 16×
动作 tokenizer：将连续动作离散化为 256 个 bin（共 7 个 token：3 位置 + 3 角度 + 1 夹爪状态）Action tokenizer: continuous actions are discretized into 256 bins (7 tokens in total: 3 for position + 3 for rotation + 1 for gripper state)
文本 tokenizer：BPE，词表大小 65,536Text tokenizer: BPE, vocabulary size 65,536

训练配置Training Configuration

动作模型输入：M=2 帧历史图像Action model input: M=2 history image frames
Action chunk 大小：K=10（长任务）或 K=5（其他任务）Action chunk size: K=10 (long tasks) or K=5 (other tasks)
世界模型预测轮数：N=1World model prediction rounds: N=1
世界模型损失权重：α=0.04World model loss weight: α=0.04

关键创新：Attention Mask 策略Key Innovation: Attention Mask Strategy

论文针对 action chunk 自回归生成中的误差累积问题，提出选择性 attention mask：在生成当前动作 token 时，遮盖（mask）同一 chunk 中此前已生成的动作 token，使每个动作仅依赖视觉和文本输入，而非依赖前序动作。To tackle error accumulation in the autoregressive generation of action chunks, the paper proposes a selective attention mask: when the current action token is generated, the previously generated action tokens within the same chunk are masked, so that each action depends only on the visual and text inputs rather than on the preceding actions.

Attention Mask 配置对比 — **Figure 3**：三种 attention mask 配置。(a) 默认因果 mask 的动作模型；(b) 本文提出的、对前序动作选择性遮盖的动作模型；(c) 世界模型沿用标准因果 attention。**Figure 3**: Three attention mask configurations. (a) Action model with the default causal mask; (b) the action model proposed in this paper, which selectively masks the preceding actions; (c) the world model, which keeps standard causal attention.

动作与世界模型的相互增强Mutual Enhancement of the Action Model and the World Model

论文从两个维度验证互利关系：（1）加入世界模型预测任务后，动作模型的抓取成功率提升 4%，说明预测未来视觉状态有助于学习更优的动作策略；（2）世界模型以动作为条件（action world model）相比无动作条件的纯世界模型，在 LIBERO 数据集上 FVD 降低约 10%（50 帧评估），说明动作信息改善了未来帧预测质量。The paper verifies the mutual benefit along two dimensions: (1) after the world-model prediction task is added, the grasping success rate of the action model improves by 4%, showing that predicting future visual states helps learn better action policies; (2) conditioning the world model on actions (action world model) lowers FVD by about 10% on the LIBERO dataset compared with an action-free world model (50-frame evaluation), showing that action information improves the quality of future-frame prediction.

03 实验Experiments

主要在 LIBERO 基准上评估，涵盖 Spatial、Object、Goal、Long 四个子任务，指标为成功率（SR）。同时在 LIBERO 上评估世界模型质量（FVD、LPIPS）。Evaluation is carried out mainly on the LIBERO benchmark, covering the four sub-tasks Spatial, Object, Goal, and Long, with success rate (SR) as the metric. World model quality (FVD, LPIPS) is also evaluated on LIBERO.

Table 2：动作模型性能对比（LIBERO）Table 2: Action Model Performance Comparison (LIBERO)

Model	Spatial SR	Object SR	Goal SR	Long SR	Average SR
OpenVLA（基线）OpenVLA (baseline)	84.7%	88.4%	79.2%	53.7%	76.5%
WorldVLA 256×256	85.6%	89.0%	82.6%	59.0%	79.1%
WorldVLA 512×512	87.6%	96.2%	83.4%	60.0%	81.8%

Table 4：世界模型生成质量（LIBERO）Table 4: World Model Generation Quality (LIBERO)

Model	FVD↓（10 帧）FVD↓ (10 frames)	FVD↓（50 帧）FVD↓ (50 frames)	LPIPS↓（10 帧）LPIPS↓ (10 frames)	LPIPS↓（50 帧）LPIPS↓ (50 frames)
纯世界模型（World Model only）World Model only	250.0	718.6	11.97	15.60
动作世界模型（Action World Model）Action World Model	255.1	674.1	11.94	15.44

定性结果对比 — **Figure 4**：定性可视化对比。上方为纯动作模型的执行结果，下方为 WorldVLA（动作世界模型）的执行结果，展示世界模型集成如何改善抓取任务的成功率。**Figure 4**: Qualitative visual comparison. The top row shows rollouts of the action-only model and the bottom row those of WorldVLA (the action world model), illustrating how integrating the world model improves the success rate of grasping tasks.

消融实验（Ablation）Ablation Study

Table 3 分析了各组件对动作模型性能的影响：Table 3 analyzes the impact of each component on action model performance:

配置Configuration	Goal SR	Object SR	Spatial SR	Long SR	Average SR
仅动作（Action only）Action only	67.3%	82.9%	77.8%	23.0%	62.8%
+世界模型（+World model）+World model	73.1%	88.0%	80.2%	27.3%	67.2%
+Action chunking（无 mask）+Action chunking (no mask)	79.6%	82.9%	36.7%	16.9%	54.0%
+Action chunking + attention mask	84.4%	90.9%	81.8%	49.3%	76.6%
完整模型（Full model）Full model	85.1%	90.9%	84.0%	52.4%	78.1%

关键发现：仅加入 action chunking（不加 mask）会导致平均 SR 从 67.2% 骤降至 54.0%（尤其 Spatial SR 从 80.2% 跌至 36.7%）；加入 attention mask 后恢复至 76.6%，再结合世界模型提升至 78.1%。Key finding: adding action chunking alone (without the mask) makes the average SR plunge from 67.2% to 54.0% (Spatial SR in particular falls from 80.2% to 36.7%); adding the attention mask restores it to 76.6%, and combining it with the world model raises it further to 78.1%.

Chunk 长度消融 — **Figure 6**：动作块长度消融实验。展示在不同 chunk size 下，朴素自回归与本文 attention mask 策略之间的性能对比。**Figure 6**: Ablation on action chunk length. It shows the performance comparison between naive autoregression and the attention mask strategy proposed in this paper under different chunk sizes.

历史帧数量分析（Table 5）Analysis of the Number of History Frames (Table 5)

在启用 action chunking 的条件下：1 帧输入 SR 为 74.0%；2 帧输入 SR 为 84.4%；4 帧输入 SR 为 84.7%。选用 2 帧作为默认配置，在性能与计算效率之间取得最优平衡。With action chunking enabled: 1-frame input yields an SR of 74.0%; 2-frame input yields 84.4%; 4-frame input yields 84.7%. Two frames are adopted as the default configuration, striking the best balance between performance and computational efficiency.

04 局限性Limitations

Note：以下局限性均为论文作者在 Future Work 小节中明确陈述（stated），非推断。Note: All limitations below are explicitly stated by the authors in the Future Work section, rather than inferred.

数据与模型规模尚未充分探索Scaling of data and model size not yet fully explored

论文指出 "scaling of both data and model size emerges as a promising avenue"，当前实验规模有限，更大规模训练的潜力尚待验证。The paper notes that "scaling of both data and model size emerges as a promising avenue"; the current experiments are limited in scale, and the potential of larger-scale training remains to be verified.

离散图像 tokenizer 的感知表达受限Limited perceptual expressiveness of the discrete image tokenizer

当前使用的离散图像 tokenizer "exhibits limitations in perceptual expressiveness"，作者建议未来开发 "a unified tokenizer capable of both understanding and generating high-quality visual content"。The discrete image tokenizer currently used "exhibits limitations in perceptual expressiveness", and the authors suggest developing in future work "a unified tokenizer capable of both understanding and generating high-quality visual content".

辅助动作头（auxiliary action head）尚未引入An auxiliary action head has not yet been introduced

论文认为引入 "an auxiliary action head" 有助于进一步增强抓取能力，但当前版本尚未实现。The paper argues that introducing "an auxiliary action head" would help further strengthen grasping capability, but the current version does not yet implement it.

评估局限于 LIBERO 仿真环境Evaluation limited to the LIBERO simulation environment

所有定量实验均在 LIBERO 仿真基准上进行，未在真实机器人硬件上验证泛化性能。（inferred from scope）All quantitative experiments are conducted on the LIBERO simulation benchmark; generalization has not been validated on real robot hardware. (inferred from scope)