机器人 · Embodied AI · arXiv 2024Robotics · Embodied AI · arXiv 2024
3D-VLA
A 3D Vision-Language-Action Generative World Model
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan · UMass Amherst & MIT
3D-VLA 将三维感知、语言推理与机器人动作通过生成式世界模型无缝连接：不仅能回答空间问题、定位目标，还能生成RGB-D目标图像和点云来指导机器人规划，首次在统一框架内同时覆盖感知、生成与执行三大能力。3D-VLA seamlessly connects 3D perception, language reasoning and robot action through a generative world model: it can answer spatial questions and localize targets, and it can also generate RGB-D goal images and point clouds to guide robot planning, covering perception, generation and execution within one unified framework for the first time.
3D-VLAembodied world model视觉-语言-动作vision-language-actionrobot manipulationgoal generation3D perception机器人操作robot manipulationfoundation model
01动机Motivation
现有视觉-语言-动作（VLA）模型依赖二维输入，无法充分理解物理世界的三维结构；同时，它们直接从感知映射到动作，缺乏对世界动态的更广泛理解——即世界模型能力的缺失。Existing vision-language-action (VLA) models rely on 2D inputs and cannot fully understand the 3D structure of the physical world; at the same time they map directly from perception to action, lacking a broader understanding of world dynamics, that is, the absence of a world model capability.
"Current embodied models learn a direct mapping from perception to action, devoid of a broader understanding of the dynamics of the world."
人类在行动前会在脑海中想象执行结果：拿起一杯水后，杯子的位置会如何变化？3D-VLA 正是赋予模型这种想象力——通过生成目标图像和目标点云来显式建模操作后的场景变化，进而指导机器人规划。Before acting, humans imagine the outcome in their mind: after picking up a cup of water, how will the position of the cup change? 3D-VLA endows the model with exactly this kind of imagination, explicitly modelling the change of the scene after manipulation by generating goal images and goal point clouds, and thereby guiding robot planning.
图1：3D Embodied Instruction Tuning Dataset 的样本示例，涵盖多种机器人操作场景。数据集包含来自12个机器人数据集和人-物交互数据集的 316k 条episode、共 2M 个3D-language-action 数据对，支持具身问答、任务描述、目标定位与动作规划等任务类型。Figure 1: Sample cases of the 3D Embodied Instruction Tuning Dataset, covering a variety of robot manipulation scenarios. The dataset contains 316k episodes and 2M 3D-language-action pairs drawn from 12 robot datasets and human-object interaction datasets, and supports task types such as embodied question answering, task captioning, goal localization and action planning.
68%RLBench Put Knife 成功率RLBench Put Knife success rate
02方法Method
3D-VLA 基于 BLIP2-FlanT5XL 构建，通过引入交互token（interaction tokens）扩展语言模型的表达空间，并对接预训练的具身扩散模型（embodied diffusion models）实现多模态目标生成，最终输出离散化的7-DoF机器人动作序列。3D-VLA is built on BLIP2-FlanT5XL. It extends the expressive space of the language model by introducing interaction tokens, and couples pre-trained embodied diffusion models to achieve multimodal goal generation, finally emitting discretized 7-DoF robot action sequences.
图2：3D-VLA 整体 pipeline。输入为RGB-D图像和自然语言指令，经过3D-LLM骨干网络（Q-Former + FlanT5）进行推理，通过特殊目标生成token触发具身扩散模型生成RGB-D目标图像或目标点云，同时输出离散化的7-DoF动作token序列。Figure 2: The overall 3D-VLA pipeline. The inputs are an RGB-D image and a natural language instruction; reasoning is performed by the 3D-LLM backbone (Q-Former + FlanT5); special goal-generation tokens trigger the embodied diffusion models to produce an RGB-D goal image or a goal point cloud, while a discretized 7-DoF action token sequence is emitted at the same time.
交互 Token（Interaction Tokens）Interaction Tokens
为了使语言模型能够理解并操作3D空间信息，3D-VLA 引入了四类特殊token：In order to let the language model understand and manipulate 3D spatial information, 3D-VLA introduces four categories of special tokens:
<obj> 对象token：标注句子中被操作的物体名词，如 <obj>a chocolate bar</obj>[loc tokens]。<obj> object token: marks the noun of the manipulated object in the sentence, e.g. <obj>a chocolate bar</obj>[loc tokens].
<loc0-255> 位置token：6个离散token表示物体的轴对齐3D包围盒（AABB），实现精确的空间定位。<loc0-255> location token: 6 discrete tokens represent the axis-aligned 3D bounding box (AABB) of the object, achieving precise spatial grounding.
<scene> 场景token：封装静态场景嵌入，支持模型理解完整的3D场景上下文。<scene> scene token: encapsulates the static scene embedding, supporting the model in understanding the complete 3D scene context.
动作token（Action Tokens）：将7-DoF机器人控制离散化为 <aloc0-255>（手臂位置）、<arot0-255>（旋转）、<gripper0/1>（夹爪状态）三组token，以 <ACT_SEP> 分隔。Action Tokens: discretize 7-DoF robot control into three groups of tokens, <aloc0-255> (arm position), <arot0-255> (rotation) and <gripper0/1> (gripper state), separated by <ACT_SEP>.
3D-VLA 预训练了两个扩散模型以支持多模态目标生成：3D-VLA pre-trains two diffusion models to support multimodal goal generation:
RGB-D 目标图像生成RGB-D Goal Image Generation
输入当前RGB-D图像与操作指令，生成操作执行后的目标RGB-D图像。模型通过 <image></image> token触发，经transformer投影仪对接预训练扩散解码器。Given the current RGB-D image and the manipulation instruction, the model generates the goal RGB-D image after the manipulation has been executed. Generation is triggered by the <image></image> token and is connected to the pre-trained diffusion decoder through a transformer projector.
点云目标生成Point Cloud Goal Generation
生成操作后场景的目标点云分布。模型通过 <pcd></pcd> token触发，解码为结构化的三维点云表示，用于基于点云的规划器。The model generates the goal point cloud distribution of the scene after manipulation. It is triggered by the <pcd></pcd> token and decoded into a structured 3D point cloud representation, used by point-cloud-based planners.
从12个机器人操控数据集（Open-X Embodiment）和人-物交互数据集（Epic-Kitchens、HOI4D）中构建。深度图像通过 ZoeDepth 估计，光流辅助精炼点云，3D包围盒自动提取，语言多样化借助 ChatGPT 实现。最终获得 316k episodes、2M 3D-language-action 数据对，涵盖具身问答、任务描述、目标定位、目标生成和动作规划五类任务。训练分两阶段：预训练（6×V100 32GB，30 epochs）和对齐（6×V100 64GB，20 epochs）。Built from 12 robot manipulation datasets (Open-X Embodiment) and human-object interaction datasets (Epic-Kitchens, HOI4D). Depth images are estimated with ZoeDepth, optical flow assists in refining the point clouds, 3D bounding boxes are extracted automatically, and language diversification is realized with ChatGPT. The final result is 316k episodes and 2M 3D-language-action pairs, covering five task types: embodied question answering, task captioning, goal localization, goal generation and action planning. Training proceeds in two stages: pre-training (6×V100 32GB, 30 epochs) and alignment (6×V100 64GB, 20 epochs).
03实验Experiments
在三大任务维度进行系统评测：3D推理与定位、多模态目标生成、具身动作规划。基线涵盖 BLIP2 FlanT5XL、CoVLM、Instruct-P2P、Point-E、LanCon-Learn、MCIL 等。Systematic evaluation is carried out along three task dimensions: 3D reasoning and localization, multimodal goal generation, and embodied action planning. The baselines include BLIP2 FlanT5XL, CoVLM, Instruct-P2P, Point-E, LanCon-Learn and MCIL.
3D 推理与定位（Table 1 & 2）3D Reasoning and Localization (Table 1 & 2)
图3：RGB-D 目标图像生成可视化。第1行来自测试集，第2行来自未见环境（unseen environments）。3D-VLA 能够准确预测操作执行后的场景状态，为机器人提供可视化的规划目标。Figure 3: Visualization of RGB-D goal image generation. Row 1 comes from the test set and row 2 from unseen environments. 3D-VLA accurately predicts the state of the scene after the manipulation has been executed, providing the robot with a visual planning goal.
图4：RLBench 上的RGB-D目标图像与点云可视化（上），以及 CALVIN 长程规划设置下的任务完成情况（下）。在CALVIN基准上，3D-VLA 的单任务成功率（44.7%）和连续两任务成功率（16.3%）均显著超越 MCIL 基线（28.2% 和 2.5%）。Figure 4: RGB-D goal images and point cloud visualizations on RLBench (top), together with task completion under the CALVIN long-horizon planning setting (bottom). On the CALVIN benchmark, the success rate of 3D-VLA on a single task (44.7%) and on two consecutive tasks (16.3%) clearly surpasses the MCIL baseline (28.2% and 2.5%).
注：论文未设独立的 Limitations 章节。以下条目综合论文 Impact Statement 中明确陈述的内容（标注为「作者指出」）及从设计方案中推断的局限（标注为「推断」）。Note: the paper has no standalone Limitations section. The items below combine the points explicitly stated in the Impact Statement of the paper (marked as "stated by the authors") with limitations inferred from the design (marked as "inferred").
真实环境部署风险（作者指出）Risks of real-world deployment (stated by the authors)
论文在 Impact Statement 中明确指出：真实世界的机器人部署存在碰撞风险，需要人工监督加以缓解。模型目前主要在仿真环境（RLBench、CALVIN）中验证，真实机器人上的零样本迁移能力尚未全面评估。The paper explicitly states in its Impact Statement that real-world robot deployment carries a risk of collision and needs human supervision to mitigate it. The model is at present validated mainly in simulated environments (RLBench, CALVIN), and its zero-shot transfer ability on real robots has not been evaluated comprehensively.
开环控制假设（推断）Open-loop control assumption (inferred)
动作预测采用开环控制（open-loop control），模型不利用执行过程中的观察历史来修正动作序列。对比基线（如 LanCon-Learn）同样基于此假设，但实际任务执行中闭环反馈更为鲁棒。Action prediction adopts open-loop control: the model does not use the observation history collected during execution to correct the action sequence. The compared baselines (such as LanCon-Learn) rest on the same assumption, yet closed-loop feedback is more robust in actual task execution.
深度估计质量限制（推断）Limits of depth estimation quality (inferred)
数据集中大量RGB-D数据由 ZoeDepth 估计获得，而非真实深度传感器采集。估计深度的噪声和域偏移可能影响3D感知精度，尤其在涉及精细操作的任务中。A large part of the RGB-D data in the dataset is obtained by ZoeDepth estimation rather than captured by a real depth sensor. The noise and domain shift of estimated depth may affect the accuracy of 3D perception, especially in tasks that involve fine-grained manipulation.
域特化训练依赖（作者指出）Dependence on domain-specific training (stated by the authors)
论文实验表明，在特定任务上进行域特化微调能显著提升性能；零样本迁移至未见环境时表现下降（如 Pick Up Cup 从 40% 降至 24%），说明模型泛化能力仍有提升空间。Experiments in the paper show that domain-specific fine-tuning on a particular task improves performance significantly; performance drops under zero-shot transfer to unseen environments (for example Pick Up Cup falls from 40% to 24%), which indicates that the generalization ability of the model still leaves room for improvement.
