VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
用大语言模型组合 3D 价值图，零样本合成机器人操控轨迹Composing 3D value maps with large language models to synthesize robotic manipulation trajectories zero-shot
Wenlong Huang · Chen Wang · Ruohan Zhang · Yunzhu Li · Jiajun Wu · Li Fei-Fei | Stanford University & UIUC
VoxPoser 利用大语言模型（LLM）的代码生成能力，通过调用视觉语言模型（VLM）在三维观测空间中组合语言条件的 affordance/avoidance value maps，再由运动规划器将 value map 作为目标函数，零样本合成六自由度闭环机器人轨迹。系统无需额外训练，可泛化至开集语言指令与开集物体。VoxPoser exploits the code-generation capability of large language models (LLM) to call vision-language models (VLM) and compose language-conditioned affordance/avoidance value maps in the three-dimensional observation space; a motion planner then takes the value map as its objective function and synthesizes closed-loop 6-DoF robot trajectories zero-shot. The system needs no additional training and generalizes to open-set language instructions and open-set objects.
robotic manipulationlarge language models3D value mapszero-shot trajectory synthesisvision-language modelsmodel-based planningcode-writingaffordance groundingclosed-loop MPC机器人操控robotic manipulation
01动机Motivation
当前基于 LLM 的机器人系统绝大多数依赖预定义的 motion primitives（如 move_to_pose、close_gripper），这构成了系统泛化能力的根本瓶颈——每新增一个技能都需要人工设计与大量数据收集。Most current LLM-based robotic systems rely on pre-defined motion primitives (such as move_to_pose and close_gripper), which forms the fundamental bottleneck of their generalization ability—every new skill demands manual design and large-scale data collection.
"Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck."
VoxPoser 提出的核心洞察是：LLM 擅长推理 affordances（目标区域）与 constraints（避障约束），且它的代码生成能力可以直接调用感知 API，在三维体素空间中构建稠密的 value map，将抽象的语言知识锚定在机器人可感知的观测空间中，从而完全绕过手工设计的 primitives。The core insight of VoxPoser is that LLMs are good at reasoning about affordances (target regions) and constraints (obstacle-avoidance requirements), and that their code-generation capability can directly call perception APIs to build dense value maps in a three-dimensional voxel space, grounding abstract linguistic knowledge in the observation space the robot can perceive and thereby bypassing hand-designed primitives entirely.
图 1（论文 Figure 1）：VoxPoser 概览。LLM 从自然语言指令中提取 affordances 与 constraints，通过 VLM 将其落地为 3D value map，运动规划器以 value map 为目标函数直接合成机器人轨迹，覆盖"挂毛巾""关抽屉""分类垃圾"等多种日常操作任务，且无需针对特定任务或 LLM 进行额外训练。Figure 1 (paper Figure 1): overview of VoxPoser. The LLM extracts affordances and constraints from the natural-language instruction and grounds them into a 3D value map through a VLM; the motion planner takes the value map as its objective function and directly synthesizes the robot trajectory, covering everyday manipulation tasks such as hanging a towel, closing a drawer and sorting trash, with no additional training for a specific task or LLM.
88%真实机器人静态场景成功率（5 类任务）Real-robot success rate in static scenes (5 task types)
70%动态扰动下真实机器人成功率Real-robot success rate under dynamic disturbances
13仿真评估任务数（SAPIEN 环境）Number of tasks in the simulation evaluation (SAPIEN environment)
2,766仿真评估中唯一指令数量Number of unique instructions in the simulation evaluation
02方法Method
VoxPoser 将语言指令映射为 3D value map，再由运动规划器合成轨迹。整个流程由 LLM 生成 Python 代码驱动，无需任何额外训练。VoxPoser maps a language instruction into a 3D value map, from which a motion planner synthesizes the trajectory. The whole pipeline is driven by Python code generated by the LLM and requires no additional training whatsoever.
方法流程图（来自项目主页）：(a) LLM 生成代码，调用 VLM 获取物体感知信息，组合多种类型的 3D value map；(b) 运动规划器以 value map 为目标函数，通过 greedy search 合成 6-DoF 轨迹，并以 5 Hz 频率闭环重规划。Method pipeline (from the project page): (a) the LLM generates code that calls the VLM to obtain object perception information and composes several types of 3D value map; (b) the motion planner takes the value map as its objective function and synthesizes a 6-DoF trajectory by greedy search, replanning in closed loop at 5 Hz.
Language Model Programs（LMP）与 3D Value Map 组合Language Model Programs (LMP) and 3D Value Map Composition
VoxPoser 定义了多种类型的 value map，每种由一个专用 LMP 负责生成，每个 LMP 接收自然语言子任务描述，输出形状为 (100, 100, 100, k) 的体素 map：VoxPoser defines several types of value map, each generated by a dedicated LMP; every LMP takes a natural-language sub-task description and outputs a voxel map of shape (100, 100, 100, k):
Affordance map（k=1）：目标抓取/接触区域赋高值（如抽屉把手）。Affordance map (k=1): assigns high values to the target grasping/contact region (e.g. a drawer handle).
Avoidance map（k=1）：危险区域赋低值（如花瓶周围）；对 avoidance map 施加 Gaussian filter。Avoidance map (k=1): assigns low values to hazardous regions (e.g. around a vase); a Gaussian filter is applied to the avoidance map.
Planner LMP：将高层指令 L 分解为子任务序列 ℓ₁:N。Planner LMP: decomposes the high-level instruction L into a sub-task sequence ℓ₁:N.
Composer LMP：接收子任务 ℓᵢ，按需调用上述各 map LMP 并传入细粒度语言参数化。Composer LMP: takes a sub-task ℓᵢ and calls the map LMPs above as needed, passing in fine-grained language parameterization.
规划器将 affordance map 权重设为 2、avoidance map 权重设为 1，对归一化后的加权和取负作为 cost map，用 greedy search 找到无碰撞的末端执行器位置序列 p₁:N ∈ ℝ³；rotation、velocity、gripper map 在各 waypoint 处单独施加约束。6-DoF 轨迹确定后执行第一个 waypoint，随即以 5 Hz 频率基于最新观测重规划，实现对动态扰动的鲁棒性。The planner sets the affordance map weight to 2 and the avoidance map weight to 1, negates the normalized weighted sum as the cost map, and uses greedy search to find a collision-free sequence of end-effector positions p₁:N ∈ ℝ³; the rotation, velocity and gripper maps impose their constraints separately at each waypoint. Once the 6-DoF trajectory is determined the first waypoint is executed, and the planner immediately replans from the latest observation at 5 Hz, achieving robustness to dynamic disturbances.
接触丰富任务的在线 Dynamics 学习Online Dynamics Learning for Contact-Rich Tasks
对于推门、开冰箱等 contact-rich 任务，零样本轨迹作为探索先验，驱动对环境动力学模型（平面推动模型：接触点、推动方向与距离）的高效在线学习，并用 MPC + random shooting 优化动作参数。For contact-rich tasks such as pushing a door open or opening a fridge, the zero-shot trajectory serves as an exploration prior that drives efficient online learning of an environment dynamics model (a planar pushing model: contact point, pushing direction and distance), with the action parameters optimized by MPC + random shooting.
论文 Figure 2：详细流程示例——给定指令"Open the top drawer, and watch out for that vase"，LLM 生成 Python 代码依次调用感知 API，构建 affordance（抽屉把手区域）与 avoidance（花瓶周围）两类 map，最终由规划器合成规避花瓶同时操作抽屉的轨迹。Paper Figure 2: a detailed pipeline example—given the instruction "Open the top drawer, and watch out for that vase", the LLM generates Python code that calls the perception APIs in turn to build two kinds of map, affordance (the drawer-handle region) and avoidance (around the vase); the planner finally synthesizes a trajectory that avoids the vase while operating the drawer.论文 Emergent Capabilities 图：系统的三种涌现能力——(左) 行为常识推理（如理解"左撇子"语境）；(中) 基于用户反馈的细粒度语言纠错；(右) 多步视觉程序，适应物体几何信息不足的情况。Paper Emergent Capabilities figure: three emergent capabilities of the system—(left) behavioral commonsense reasoning (such as understanding a left-handed context); (middle) fine-grained language correction from user feedback; (right) multi-step visual programs that adapt when object geometry information is insufficient.
03实验Experiments
评估在两个环境下进行：真实 Franka Emika Panda 机器人（双 Azure Kinect RGB-D 摄像头）完成 5 类日常操作任务；SAPIEN 仿真平台上评估 13 个高度随机化的任务，共 2,766 条唯一指令。基线：LLM + Primitives（Code as Policies 变体，使用 GPT-4 参数化预定义 primitives）；U-Net + Motion Planning（有监督学习 2D costmap）。The evaluation is carried out in two environments: a real Franka Emika Panda robot (two Azure Kinect RGB-D cameras) performing 5 everyday manipulation tasks, and the SAPIEN simulation platform, on which 13 highly randomized tasks are evaluated with 2,766 unique instructions in total. Baselines: LLM + Primitives (a Code as Policies variant that uses GPT-4 to parameterize pre-defined primitives); U-Net + Motion Planning (supervised learning of a 2D costmap).
表 1：真实机器人实验（10 次/任务）Table 1: real-robot experiments (10 trials per task)
论文第 7 页：包含真实机器人实验（Table 1）与仿真泛化实验（Table 2）的完整结果，以及错误来源分析（Figure 4）——VoxPoser 显著降低了"specification error"，主要失败来自感知模块（OWL-ViT 对物体初始姿态敏感）。Page 7 of the paper: the full results of the real-robot experiments (Table 1) and the simulation generalization experiments (Table 2), together with the error-source analysis (Figure 4)—VoxPoser markedly reduces "specification error", and the main failures come from the perception module (OWL-ViT is sensitive to the initial pose of objects).
消融与错误分析Ablations and Error Analysis
错误分析（Figure 4）显示，VoxPoser 相比基线大幅减少了 specification error（即规划意图与实际执行不符的错误），真实机器人的大多数失败案例归因于感知模块对物体初始姿态的敏感性，而非方法本身的规划能力。仿真中对 seen 与 unseen 指令/属性的表现相近，表明泛化能力来自 LLM 的开放世界知识，而非对特定训练分布的记忆。The error analysis (Figure 4) shows that VoxPoser greatly reduces specification error (errors in which the planned intent does not match the actual execution) compared with the baselines; most failure cases on the real robot are attributed to the perception module's sensitivity to the initial pose of objects rather than to the planning ability of the method itself. In simulation, performance on seen and unseen instructions/attributes is comparable, indicating that the generalization comes from the LLM's open-world knowledge rather than from memorizing a particular training distribution.
04局限性Limitations
Note:以下限制均为作者在论文"Conclusion, Limitations, & Future Works"一节中明确陈述。All of the limitations below are explicitly stated by the authors in the paper's "Conclusion, Limitations, & Future Works" section.
依赖外部感知模块，难以处理精细几何Reliance on external perception modules; fine-grained geometry is hard to handle
"It relies on external perception modules, which is limiting in tasks that require holistic visual reasoning or understanding of fine-grained object geometries."——当任务需要整体场景理解或精细物体几何时，基于 bounding box + 点云的感知链路存在明显短板。"It relies on external perception modules, which is limiting in tasks that require holistic visual reasoning or understanding of fine-grained object geometries."—when a task requires holistic scene understanding or fine-grained object geometry, a perception chain based on bounding boxes plus point clouds has clear shortcomings.
Contact-rich 任务仍需通用 Dynamics 模型Contact-rich tasks still require a general-purpose dynamics model
"While applicable to efficient dynamics learning, a general-purpose dynamics model is still required to achieve contact-rich tasks with the same level of generalization."——论文中的 dynamics 学习仅针对平面推动模型，要实现与零样本任务同等泛化能力，仍需更通用的物理模型。"While applicable to efficient dynamics learning, a general-purpose dynamics model is still required to achieve contact-rich tasks with the same level of generalization."—the dynamics learning in the paper covers only a planar pushing model; achieving the same level of generalization as the zero-shot tasks still requires a more general physical model.
仅规划末端执行器轨迹，未考虑全臂规划Plans end-effector trajectories only, without whole-arm planning
"Our motion planner considers only end-effector trajectories while whole-arm planning is also feasible and likely a better design choice."——当前规划器忽略机械臂本体碰撞，全臂规划将是更优但更复杂的选择。"Our motion planner considers only end-effector trajectories while whole-arm planning is also feasible and likely a better design choice."—the current planner ignores collisions of the arm body itself; whole-arm planning would be a better but more complex choice.
需要手动 Prompt EngineeringManual prompt engineering is required
"Manual prompt engineering is required for LLMs."——每个 LMP 需要 5–20 条精心设计的示例 query-response 对，在部署到新机器人平台或新任务领域时增加了适配成本。"Manual prompt engineering is required for LLMs."—each LMP needs 5–20 carefully designed example query-response pairs, which adds an adaptation cost when deploying to a new robot platform or a new task domain.
