DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

01 动机Motivation

当前 VLA 模型大多直接将视觉观测与语言指令映射为动作序列，缺乏对"未来会发生什么"的显式推理。已有研究尝试通过预测整帧图像或生成子目标来引入前瞻性，但这类方法引入了大量冗余像素信息、缺少显式 3D 知识，且对未来状态的高层理解十分有限，导致泛化能力不足。Most current VLA models map visual observations and language instructions directly into action sequences, lacking explicit reasoning about "what will happen next". Existing work tries to introduce foresight by predicting whole future frames or generating sub-goals, but such methods bring in a large amount of redundant pixel information, lack explicit 3D knowledge, and offer only very limited high-level understanding of future states, which leaves generalization insufficient.

"Prior work relies on redundant pixel information, lacks explicit 3D knowledge, and fails to capture high-level understanding of future states."

VLA方法对比示意图 — **图 1：**四类 VLA 范式对比。(a) 普通 VLA 直接映射观测→动作；(b) 借助独立图像/视频生成或 copilot 模型预测未来帧/轨迹后再驱动动作头；(c) 先预测子目标图像再生成动作；(d) 本文提出的 DreamVLA，显式预测动态区域、深度图与语义特征（DINOv2 + SAM）作为世界知识，显著增强动作推理与泛化。**Figure 1:** Comparison of four VLA paradigms. (a) A plain VLA maps observations directly to actions; (b) a separate image/video generation or copilot model predicts future frames/trajectories that then drive the action head; (c) a sub-goal image is predicted first and actions are generated afterwards; (d) DreamVLA, proposed in this paper, explicitly predicts dynamic regions, depth maps and semantic features (DINOv2 + SAM) as world knowledge, markedly strengthening action reasoning and generalization.

4.44CALVIN ABC-D 平均长度（SOTA）average length on CALVIN ABC-D (SOTA)

92.6%LIBERO 综合成功率overall success rate on LIBERO

76.7%真实 Franka 机器人成功率success rate on a real Franka robot

3×世界知识类型（动态 + 深度 + 语义）types of world knowledge (dynamic + depth + semantic)

02 方法Method

DreamVLA 将多模态输入（RGB 图像、语言指令、机器人状态）送入统一的 GPT-2 骨干 Transformer，通过两类可学习查询令牌——<dream>（世界知识预测）与 <action>（动作生成）——同步建模"下一步世界状态"和"下一步执行动作"。预测分支通过三个轻量解码器分别输出动态区域、深度图和语义特征，动作分支则使用 diffusion transformer 生成连续动作序列。DreamVLA feeds multimodal inputs (RGB images, language instructions, robot state) into a unified GPT-2 backbone Transformer and, through two kinds of learnable query tokens — <dream> (world-knowledge prediction) and <action> (action generation) — jointly models "the next world state" and "the next action to execute". The prediction branch emits dynamic regions, depth maps and semantic features through three lightweight decoders, while the action branch uses a diffusion transformer to generate continuous action sequences.

DreamVLA框架总览 — **图 2：DreamVLA 整体框架。**多模态输入经各自编码器（CLIP 文本编码器、Masked Autoencoder 视觉编码器、状态编码器）嵌入后，拼接送入 LLM；LLM 输出分两路：`<dream>` 查询经三个轻量解码器分别预测动态区域（CoTracker 监督）、深度图（Depth-Anything 监督）、语义特征（DINOv2 + SAM 监督）；`<action>` 查询送入 diffusion transformer 生成最终动作。**Figure 2: Overall framework of DreamVLA.** Multimodal inputs are embedded by their respective encoders (CLIP text encoder, Masked Autoencoder visual encoder, state encoder), then concatenated and fed into the LLM; the LLM output splits into two paths: the `<dream>` queries pass through three lightweight decoders that predict dynamic regions (supervised by CoTracker), depth maps (supervised by Depth-Anything) and semantic features (supervised by DINOv2 + SAM); the `<action>` queries are fed into a diffusion transformer to generate the final actions.

动态区域预测（Dynamic Regions）Dynamic Region Prediction (Dynamic Regions)

DreamVLA 并不预测稠密光流，也不合成整帧未来图像，而是利用 CoTracker 提取"动态区域"——即随机器人末端执行器或其他可移动物体运动的像素集合，仅在这些区域上进行重建训练。这一设计大幅降低了无关背景像素的干扰："predicting dynamic regions alone delivers the greatest gains."DreamVLA neither predicts dense optical flow nor synthesizes whole future frames; instead it uses CoTracker to extract "dynamic regions" — the set of pixels that move with the robot end-effector or with other movable objects — and performs reconstruction training only on those regions. This design substantially reduces interference from irrelevant background pixels: "predicting dynamic regions alone delivers the greatest gains."

动态区域可视化 — **图 3：动态区域随时间的变化可视化。**左列为固定视角相机，右列为腕部相机。蓝色掩码标出在操作过程中发生运动的区域，随任务进程而动态更新。**Figure 3: Visualization of how dynamic regions evolve over time.** The left column is the fixed-view camera, the right column the wrist camera. Blue masks mark the regions that move during manipulation, updated dynamically as the task proceeds.

深度预测与语义特征预测Depth Prediction and Semantic Feature Prediction

Depth Forecasting

使用 Depth-Anything 生成深度监督信号，损失函数为尺度归一化 MSE（scale-normalized MSE loss）。预测未来深度场告知机器人下一步应移向何处、远离哪些潜在障碍物，为规划提供显式 3D 知识。Depth-Anything is used to generate the depth supervision signal, with a scale-normalized MSE loss. Predicting the future depth field tells the robot where it should move next and which potential obstacles to stay away from, providing explicit 3D knowledge for planning.

Semantic Features（DINOv2 + SAM）

结合 DINOv2 和 SAM 的特征，通过对比学习 InfoNCE loss 进行监督。语义预测教会模型"哪些物体/区域对任务最重要"，提供目标物体身份与可操作性等高层上下文，引导目标选择与抓取决策。Features from DINOv2 and SAM are combined and supervised by a contrastive InfoNCE loss. Semantic prediction teaches the model "which objects/regions matter most for the task", supplying high-level context such as target-object identity and affordance, and thereby guiding object selection and grasping decisions.

Block-wise Structured Attention

为防止不同查询类型之间的信息泄露，DreamVLA 在自注意力层中引入分块掩码机制（block-wise structured attention）。如图 4 所示，<dream> 查询与 <action> 查询各自只能与上下文 token 及同类查询交互，而不能相互"窥视"，从而确保每个预测头的训练信号干净且信息不对称。消融实验显示，去除该机制使平均长度从 4.44 降至 3.75（下降 0.69）。To prevent information leakage between the different query types, DreamVLA introduces a block-wise masking mechanism (block-wise structured attention) in the self-attention layers. As shown in Figure 4, the <dream> queries and the <action> queries can each interact only with the context tokens and with queries of the same kind, and cannot "peek" at one another, which keeps the training signal of every prediction head clean and the information asymmetric. Ablations show that removing this mechanism drops the average length from 4.44 to 3.75 (a decrease of 0.69).

分块结构化注意力掩码 — **图 4：Block-wise Structured Attention。**注意力掩码将 context tokens、dream queries 和 action queries 三类 token 的交互严格隔离，防止跨模态信息提前泄露，保证各分支梯度信号独立清晰。**Figure 4: Block-wise Structured Attention.** The attention mask strictly isolates the interactions among the three token types — context tokens, dream queries and action queries — preventing cross-modal information from leaking ahead of time and keeping the gradient signal of each branch independent and clean.

03 实验Experiments

在三个基准上评估：CALVIN ABC-D（语言条件多任务操作，5 步连续评估）、LIBERO（Spatial / Object / Goal / Long 四个子集）以及真实 Franka Panda 机器人（pick、place、drawer 三类任务）。Evaluation is carried out on three benchmarks: CALVIN ABC-D (language-conditioned multi-task manipulation, evaluated over 5 consecutive steps), LIBERO (the four subsets Spatial / Object / Goal / Long) and a real Franka Panda robot (the three task types pick, place and drawer).

CALVIN ABC-D 对比结果CALVIN ABC-D Comparison

方法Method	T1	T2	T3	T4	T5	Avg. Len.
Roboflamingo	82.4	61.9	46.6	33.1	23.5	2.47
Susie	87.0	69.0	49.0	38.0	26.0	2.69
GR-1	85.4	71.2	59.6	49.7	40.1	3.06
3D Diffusor Actor	92.2	78.7	63.9	51.2	41.2	3.27
OpenVLA	91.3	77.8	62.0	52.1	43.5	3.27
RoboDual	94.4	82.7	72.1	62.4	54.4	3.66
UNIVLA	95.5	85.8	75.4	66.9	56.5	3.80
Pi0	93.8	85.0	76.7	68.1	59.9	3.92
UP-VLA	92.8	86.5	81.5	76.9	69.9	4.08
Robovlm	98.0	93.6	85.4	77.8	70.4	4.25
Seer	96.3	91.6	86.1	80.3	74.0	4.28
VPP	95.7	91.2	86.3	81.0	75.0	4.29
DreamVLA（本文）DreamVLA (ours)	98.2	94.6	89.5	83.4	78.1	4.44

LIBERO 对比结果LIBERO Comparison

方法Method	Spatial	Object	Goal	Long	Average
Diffusion Policy	78.3	92.5	68.3	50.5	72.4
Octo	78.9	85.7	84.6	51.1	75.1
OpenVLA	84.7	88.4	79.2	53.7	76.5
SpatialVLA	88.2	89.9	78.6	55.5	78.1
CoT-VLA	81.1	87.5	91.6	87.6	69.0
DreamVLA（本文）DreamVLA (ours)	97.5	94.0	89.5	89.5	92.6

消融实验Ablation Studies

知识组合消融实验结果 — **图 6：不同世界知识组合下的 CALVIN ABC-D 性能。**All = 全部五种模型组合；All-X = 从 All 中去掉 X。可见各类知识均有贡献，其中动态区域贡献最大，深度与语义贡献相近且互补。**Figure 6: CALVIN ABC-D performance under different combinations of world knowledge.** All = the combination of all five models; All-X = All with X removed. Every kind of knowledge contributes, dynamic regions contributing the most, while depth and semantics contribute comparably and complementarily.

关键消融结论（均在 CALVIN ABC-D 上测量 Avg. Len.）：Key ablation findings (all measured as Avg. Len. on CALVIN ABC-D):

预测 vs. 重建：预测未来状态（4.44）显著优于仅做辅助重建（4.14），说明前瞻性监督是核心。Prediction vs. reconstruction: predicting future states (4.44) is markedly better than auxiliary reconstruction alone (4.14), showing that forward-looking supervision is the core.
动态区域 vs. 光流：动态区域掩码（4.44）优于稠密光流（4.23），更聚焦、噪声更少。Dynamic regions vs. optical flow: the dynamic-region mask (4.44) beats dense optical flow (4.23) — more focused and less noisy.
Structured Attention vs. 普通因果注意力：结构化掩码（4.44）vs. 普通 causal attention（3.75），差距最大，防止信息泄露至关重要。Structured Attention vs. plain causal attention: the structured mask (4.44) vs. plain causal attention (3.75), the largest gap of all, showing that preventing information leakage is essential.
分离查询 vs. 共享查询：分离的 dream/action 查询（4.44）优于共享查询（4.17）。Separate queries vs. shared queries: separate dream/action queries (4.44) outperform shared queries (4.17).

04 局限性Limitations

Note: 以下局限性均为作者在论文中明确陈述（stated by authors）。All the limitations below are explicitly stated by the authors in the paper.

操作类型受限：仅平行夹爪（parallel-gripper）Restricted manipulation type: parallel-gripper only

当前框架主要针对平行夹爪机器人手臂设计，尚不支持灵巧手（dexterous hand）等更复杂的末端执行器。作者计划"add dexterous-hand demonstrations with rich contact annotations"以扩展适用范围。The current framework is designed mainly for parallel-gripper robot arms and does not yet support more complex end-effectors such as dexterous hands. The authors plan to "add dexterous-hand demonstrations with rich contact annotations" in order to broaden its applicability.

感知模态单一：仅依赖 RGB 图像（RGB-centric）A single perception modality: RGB images only (RGB-centric)

模型的世界知识（动态区域、深度、语义）均基于 RGB 图像衍生，缺乏原生的 3D 点云或触觉信号。作者提出"introduce 3D point clouds and spatial information—and fuse them into volumetric world states"作为改进方向。The model’s world knowledge (dynamic regions, depth, semantics) is entirely derived from RGB images, with no native 3D point clouds or tactile signals. The authors propose to "introduce 3D point clouds and spatial information—and fuse them into volumetric world states" as a direction for improvement.

训练场景多样性有限Limited diversity of training scenes

训练数据涵盖的几何形状和材质种类有限（"trained on scenes with limited geometric and material diversity"），可能导致模型在分布外的真实环境中泛化能力不足。作者计划通过扩展数据采集与在线微调（on-policy fine-tuning）来增强长程鲁棒性。The training data covers only a limited range of geometries and materials ("trained on scenes with limited geometric and material diversity"), which may leave the model generalizing poorly in out-of-distribution real-world environments. The authors plan to strengthen long-horizon robustness by expanding data collection and by on-policy fine-tuning.