RynnVLA-002: A Unified Vision-Language-Action and World Model

01 动机Motivation

VLA 模型与 World Model 各有所长，却长期相互割裂：前者能生成动作，却缺乏对世界物理动态的内部"想象"；后者能预测未来视觉状态，却无法直接输出控制指令。这种"功能鸿沟"限制了两类模型在真实机器人场景中的潜力。VLA models and World Models each have their own strengths, yet have long remained separated from one another: the former can generate actions but lacks an internal "imagination" of the physical dynamics of the world; the latter can predict future visual states but cannot directly output control commands. This "functional gap" limits the potential of both families of models in real robot scenarios.

"RynnVLA-002 internalizes the VLA objective and the action-conditioned world-modeling objective in one Chameleon-style autoregressive backbone with a shared token space."

Teaser: VLA vs World Model vs Action World Model — **Figure 1：**左：标准 VLA 仅输出动作，无法预测下一帧图像；中：World Model 仅预测图像，无法生成动作；右：本文提出的 Action World Model (RynnVLA-002) 在同一模型内同时完成动作生成与视觉预测，两个目标在训练中相互强化。**Figure 1:** Left: a standard VLA only outputs actions and cannot predict the next image frame; middle: a World Model only predicts images and cannot generate actions; right: the Action World Model (RynnVLA-002) proposed in this paper performs both action generation and visual prediction inside a single model, and the two objectives reinforce each other during training.

97.4%LIBERO 平均成功率（无预训练）LIBERO average success rate (no pretraining)

+50%真实机器人整体成功率提升Overall success-rate gain on real robots

90.0%真实多目标 Place block 任务成功率Success rate on the real multi-object Place block task

65,536统一词汇表 token 数量Number of tokens in the unified vocabulary

现有 VLA 方法（如 RT-2）将动作仅置于输出端，模型内部无法建立对动作效果的推理；而现有 World Model（如 Genie）只预测视觉状态，缺乏直接的机器人控制能力。RynnVLA-002 的核心洞察是：VLA 的精准动作能力可以改善 World Model 的视觉一致性，World Model 的预见性又能反哺 VLA 的动作决策，两者在同一主干内协同训练即可实现"1+1>2"的效果。Existing VLA methods (such as RT-2) place actions only at the output end, so the model cannot internally reason about the effects of its actions; existing World Models (such as Genie) predict only visual states and lack direct robot control capability. The core insight of RynnVLA-002 is that the precise action capability of the VLA can improve the visual consistency of the World Model, while the foresight of the World Model in turn feeds back into the action decisions of the VLA, so that training both jointly inside one backbone yields a "1+1>2" effect.

02 方法Method

RynnVLA-002 采用共享自回归 Transformer 主干，将视觉 token（VQ-GAN 编码）、文本 token（BPE）、状态/动作 token（连续值离散化为 256 bins）统一到 65,536 大小的词汇表中，分别以两种序列格式进行训练：VLA 序列生成动作，World Model 序列预测下一帧图像。RynnVLA-002 adopts a shared autoregressive Transformer backbone that unifies visual tokens (encoded by VQ-GAN), text tokens (BPE) and state/action tokens (continuous values discretized into 256 bins) into a vocabulary of size 65,536, and is trained under two sequence formats: the VLA sequence generates actions, and the World Model sequence predicts the next image frame.

RynnVLA-002 系统总览 — **Figure 2：**RynnVLA-002 总体架构。左侧：VLA 数据流——输入语言目标、本体感知状态、历史观测图像，输出动作序列；右侧：World Model 数据流——输入历史图像帧与动作，预测下一时刻图像。两路共享同一图像 tokenizer/decoder（VQ-GAN），实现图像理解与生成的统一。**Figure 2:** Overall architecture of RynnVLA-002. Left: the VLA data flow — the language goal, the proprioceptive state and historical observation images are taken as input, and an action sequence is output; right: the World Model data flow — historical image frames and actions are taken as input, and the image at the next time step is predicted. The two paths share the same image tokenizer/decoder (VQ-GAN), unifying image understanding and generation.

统一 Token 化策略Unified Tokenization Strategy

图像通过 VQ-GAN 压缩（压缩比 16，codebook 大小 8192）编码为离散视觉 token；文本采用 BPE；连续状态与动作值均离散化为 256 bins 后编码。Images are compressed by VQ-GAN (compression ratio 16, codebook size 8192) and encoded into discrete visual tokens; text uses BPE; continuous state and action values are both discretized into 256 bins before encoding.

VLA 训练序列：VLA training sequence: {text} {state} {image-front-wrist}×M {action}×K
World Model 训练序列：World Model training sequence: {images-front-wrist} {action} {images-front-wrist} [重复 N 次]
损失函数：联合优化离散动作 loss、连续动作 loss、图像生成 loss。Loss function: jointly optimizes the discrete action loss, the continuous action loss and the image generation loss.

Action Chunk 生成：注意力掩码改进Action Chunk Generation: An Improved Attention Mask

在标准自回归生成中，动作 chunk 内的前一个动作 token 会影响后续动作 token，导致误差累积。本文提出修改注意力掩码，使当前动作仅依赖文本与视觉输入，禁止访问前序动作 token，从而显著缓解误差传播（尤其在 K=10 的长动作块中效果明显）。In standard autoregressive generation, an earlier action token inside an action chunk influences the subsequent action tokens, which leads to error accumulation. This paper proposes to modify the attention mask so that the current action depends only on the text and visual inputs and is forbidden to access preceding action tokens, which markedly mitigates error propagation (the effect is especially clear for long action chunks with K=10).

**Figure 3：**三种注意力掩码策略对比。左：默认 VLA（因果掩码，动作间存在依赖）；中：本文 VLA（动作 token 仅依赖语言与视觉）；右：World Model 掩码（图像块内双向注意力）。改进掩码有效隔离了动作间误差传播。**Figure 3:** Comparison of three attention mask strategies. Left: the default VLA (causal mask, with dependencies among actions); middle: the VLA of this paper (action tokens depend only on language and vision); right: the World Model mask (bidirectional attention inside an image block). The improved mask effectively isolates error propagation among actions.

Action Transformer Head：连续动作生成Action Transformer Head: Continuous Action Generation

在离散自回归主干之上，额外引入轻量级连续 Action Transformer，利用双向注意力并行生成平滑连续动作序列，具有以下优势：On top of the discrete autoregressive backbone, a lightweight continuous Action Transformer is additionally introduced, which uses bidirectional attention to generate smooth continuous action sequences in parallel. It offers the following advantages:

参数量更小，不易过拟合；fewer parameters and less prone to overfitting;
并行生成显著降低推理步数（48.2 Hz 对比离散方案的 2.5 Hz）；parallel generation markedly reduces the number of inference steps (48.2 Hz versus 2.5 Hz for the discrete scheme);
双向注意力保证轨迹全局平滑。bidirectional attention guarantees global smoothness of the trajectory.

03 实验Experiments

在 LIBERO 仿真基准（四个子任务：Spatial / Object / Goal / Long）和真实 LeRobot SO100 机械臂（两类 pick-and-place 任务，各约 248-249 条示范）上进行评测，与有/无预训练的多个 VLA baseline 对比。Evaluation is carried out on the LIBERO simulation benchmark (four subtasks: Spatial / Object / Goal / Long) and on a real LeRobot SO100 arm (two kinds of pick-and-place tasks, each with roughly 248-249 demonstrations), and compared against several VLA baselines with and without pretraining.

LIBERO 仿真结果LIBERO Simulation Results

方法Method	预训练Pretraining	LIBERO-Spatial	LIBERO-Object	LIBERO-Goal	LIBERO-Long	平均Average
OpenVLA	✓	84.7	88.4	79.2	53.7	76.5
π₀	✓	98.6	98.8	98.2	98.8	98.6
RynnVLA-002-Discrete	✗	91.2	97.4	93.4	91.2	93.3
RynnVLA-002-Continuous	✗	96.4	99.8	96.4	94.4	97.4

"RynnVLA-002 remains competitive with these pretrained baselines and outperforms most non-pretrained methods"，在无任何大规模机器人预训练的条件下达到 97.4% 平均成功率。"RynnVLA-002 remains competitive with these pretrained baselines and outperforms most non-pretrained methods", reaching a 97.4% average success rate without any large-scale robot pretraining.

真实机器人结果Real-Robot Results

任务Task	场景Setting	RynnVLA-002（无预训练）RynnVLA-002 (no pretraining)
Place block inside circle	单目标Single object	90.0%
Place block inside circle	多目标Multiple objects	90.0%
Place block inside circle	有干扰物With distractors	80.0%
Place strawberries into cup	多目标Multiple objects	80.0%

真实机器人任务场景 — **Figure 4：**真实世界评测场景。上行：Place block inside circle（单目标、多目标、有干扰物三种设置）；下行：Place strawberries into cup（多目标）。所有实验均使用 SO100 机械臂，每任务约 248-249 条示范数据。**Figure 4:** Real-world evaluation settings. Top row: Place block inside circle (three settings: single object, multiple objects and with distractors); bottom row: Place strawberries into cup (multiple objects). All experiments use the SO100 arm, with roughly 248-249 demonstrations per task.

消融实验：World Model 与 VLA 的互增益Ablations: Mutual Gains Between the World Model and the VLA

通过系统消融验证了核心设计的有效性：Systematic ablations verify the effectiveness of the core designs:

加入 World Model 后，离散动作成功率从 62.8%→67.2%，连续动作从 91.6%→94.6%（LIBERO 仿真）；after adding the World Model, the discrete-action success rate goes from 62.8%→67.2% and the continuous-action one from 91.6%→94.6% (LIBERO simulation);
真实机器人实验中，加入 World Model 后成功率从 30%→80%+；in the real-robot experiments, adding the World Model raises the success rate from 30%→80%+;
腕部相机对真实任务至关重要（缺失则成功率降至 0%）；the wrist camera is crucial for real-world tasks (without it the success rate drops to 0%);
World Model 预训练进一步提升：LIBERO-Long 从 23%→30.2%，LIBERO-Goal 从 67.3%→73.1%。World Model pretraining brings further gains: LIBERO-Long from 23%→30.2%, and LIBERO-Goal from 67.3%→73.1%.

World Model 改善注意力可视化 — **Figure 5：**可视化展示加入 World Model 训练后，模型对目标区域的注意力明显集中，减少了对无关背景的干扰，从而提升了动作决策的精准性。**Figure 5:** Visualization showing that after World Model training is added, the attention of the model concentrates clearly on the target region and interference from irrelevant background is reduced, thereby improving the precision of action decisions.

性能对比图 — **Figure 6（综合）：**各 baseline 与 RynnVLA-002 在 LIBERO 四个子任务上的对比，以及真实机器人六种设置下的成功率对比。RynnVLA-002 在所有真实机器人设置中取得最优整体表现。**Figure 6 (overall):** Comparison of the baselines and RynnVLA-002 on the four LIBERO subtasks, together with the success rates under the six real-robot settings. RynnVLA-002 achieves the best overall performance across all real-robot settings.

04 局限性Limitations

Note：以下局限性均为作者在论文中明确陈述（stated by the authors）。Note: all limitations below are explicitly stated by the authors in the paper.

真实评测场景覆盖有限Limited coverage of real-world evaluation settings

"Current real-world evaluation focuses on SO100 pick-and-place manipulation"，仅测试了两类抓放任务，尚未在更广泛的机器人平台（如双臂、移动机器人）或更复杂操作任务（如工具使用、精细装配）上进行验证。作者明确表示需要"more robot platforms, more sophisticated manipulation tasks"。"Current real-world evaluation focuses on SO100 pick-and-place manipulation": only two kinds of pick-and-place tasks were tested, and no validation has yet been carried out on broader robot platforms (such as bimanual or mobile robots) or on more complex manipulation tasks (such as tool use or fine assembly). The authors explicitly state the need for "more robot platforms, more sophisticated manipulation tasks".

训练成本高，单任务耗时约四天High training cost: a single task takes roughly four days

"training a single real-world task currently takes roughly four days"，计算开销较大，限制了快速迭代与大规模部署。"training a single real-world task currently takes roughly four days": the computational overhead is considerable, which limits rapid iteration and large-scale deployment.

自回归图像 token 生成效率瓶颈Efficiency bottleneck of autoregressive image-token generation

当前采用逐 token 自回归方式生成图像，推理效率受限（离散动作模式下最低仅 2.5 Hz）。作者指出未来需探索"more efficient image-token generation"方案，如并行解码或连续表示。Images are currently generated token by token in an autoregressive manner, so inference efficiency is limited (as low as 2.5 Hz in the discrete-action mode). The authors point out that future work needs to explore "more efficient image-token generation" schemes, such as parallel decoding or continuous representations.