在视频与语言空间中进行长时序机器人任务规划Long-Horizon Robot Task Planning in the Space of Video and Language
Yilun Du · Mengjiao Yang · Pete Florence · Fei Xia · Ayzaan Wahid · Brian Ichter · Pierre Sermanet · Tianhe Yu · Pieter Abbeel · Joshua B. Tenenbaum · Leslie Kaelbling · Andy Zeng · Jonathan Tompson | Google DeepMind · MIT · UC Berkeley
VLP（Video Language Planning）提出将 vision-language model（VLM）与 text-to-video model 组合，通过树搜索在视频与语言构成的联合空间中规划长时序机器人任务。VLM 同时充当策略（生成文字动作）和价值函数（评估视频进度），text-to-video model 充当动力学模型，在推理时以更多算力换取更高质量的视频计划。VLP (Video Language Planning) combines a vision-language model (VLM) with a text-to-video model and uses tree search to plan long-horizon robot tasks in the joint space of video and language. The VLM serves both as a policy (producing text actions) and as a value function (assessing video progress), while the text-to-video model serves as the dynamics model, trading additional inference-time compute for higher-quality video plans.
video language planninglong-horizon planningtree searchvision-language modeltext-to-video modelgoal-conditioned policy机器人操作robot manipulation长时序规划long-horizon planning
01动机Motivation
长时序机器人操作需要同时具备：（i）对高层语义任务的抽象推理；（ii）对低层物理动力学的准确建模。大语言模型（LLM）和视觉语言模型（VLM）善于高层推理，但局限于文字表达、难以推理物理约束；text-to-video 模型善于捕捉视觉动力学，但只能生成短时域片段。如何将两者结合以实现数百帧级的长时序视觉规划，是本文核心问题。Long-horizon robot manipulation requires both (i) abstract reasoning over high-level semantic tasks and (ii) accurate modeling of low-level physical dynamics. Large language models (LLMs) and vision-language models (VLMs) excel at high-level reasoning but are confined to textual expression and struggle to reason about physical constraints; text-to-video models capture visual dynamics well but can only generate short-horizon clips. How to combine the two to achieve hundreds-of-frames long-horizon visual planning is the central question of this paper.
"We are interested in enabling visual planning for complex long-horizon tasks in the space of generated videos and language, leveraging recent advances in large generative models pretrained on Internet-scale data."
图 1：VLP 总览。 给定当前图像观测，VLM policy（左上）生成下一步文字动作，video model（右上）将其转换为短时视频帧序列；VLM heuristic（左下）对各视频分支打分，树搜索（中）选出最优分支并递归扩展，最终输出长时视频计划；goal-conditioned policy（右下）逐帧执行动作。Figure 1: VLP overview. Given the current image observation, the VLM policy (top left) generates the next text action and the video model (top right) turns it into a short-horizon sequence of video frames; the VLM heuristic (bottom left) scores each video branch, tree search (center) selects the best branch and expands it recursively, finally yielding a long-horizon video plan; the goal-conditioned policy (bottom right) executes actions frame by frame.
98%仿真 Group-by-Color 视频计划成功率（VLP vs UniPi 4%）Video plan success rate on simulated Group-by-Color (VLP vs UniPi 4%)
92%仿真 Group-by-Color 任务执行完成率Task execution completion rate on simulated Group-by-Color
3支持的机器人硬件平台Supported robot hardware platforms
~100s计划可延伸至数百帧的长时序视频Plans extend to long-horizon videos of hundreds of frames
02方法Method
VLP 以三个模块为核心：（1）VLM as Policy——根据当前图像和语言目标生成候选文字动作；（2）Video Model as Dynamics——将文字动作转化为合成的短时视频帧；（3）VLM as Heuristic Function——预测距离目标完成还需多少步，用于剪枝。三者通过树搜索（beam search + parallel hill climbing）组合，在推理时可用更多计算换取更好的视频计划质量。VLP centers on three modules: (1) VLM as Policy — generating candidate text actions from the current image and the language goal; (2) Video Model as Dynamics — turning text actions into synthesized short-horizon video frames; (3) VLM as Heuristic Function — predicting how many steps remain until the goal is completed, used for pruning. The three are combined through tree search (beam search + parallel hill climbing), so that more compute at inference time can be traded for better video plan quality.
图 2：长时视频计划示例。 VLP 仅给定初始图像和语言目标，自主生成语言子目标序列（左侧列表）和对应视频帧序列（右侧图像），涵盖仿真 Language Table 环境（上）和真实桌面（下）两种场景。Figure 2: Examples of long-horizon video plans. Given only an initial image and a language goal, VLP autonomously generates a sequence of language subgoals (list on the left) and the corresponding video frames (images on the right), covering both the simulated Language Table environment (top) and a real tabletop (bottom).
VLM as Policy & Heuristic Function
VLP 使用 PaLM-E 作为 VLM。给定图像 x 和语言目标 g，policy πVLM(x, g) 采样出 A 个候选文字动作 a。heuristic function HVLM(x, g) 在长轨迹片段上微调，输出"距目标还需几步"的标量估计——取负值作为启发值（越接近目标值越大）。为防止动力学模型被"利用"（exploiting model dynamics），若某视频帧的启发值超过固定阈值则直接丢弃。VLP uses PaLM-E as the VLM. Given an image x and a language goal g, the policy πVLM(x, g) samples A candidate text actions a. The heuristic function HVLM(x, g) is fine-tuned on long trajectory segments and outputs a scalar estimate of "how many steps remain to the goal" — its negation is used as the heuristic value (the closer to the goal, the larger the value). To prevent the dynamics model from being exploited (exploiting model dynamics), any video frame whose heuristic value exceeds a fixed threshold is discarded outright.
Text-to-Video Model as Dynamics
给定当前帧 x 和文字动作 a，视频模型 fVM(x, a) 合成一段短时视频 x1:S，预测执行该动作后的视觉结果序列。长时计划通过自回归拼接（递归将最后一帧作为新起点）延伸至数百帧。对多相机双臂平台（ALOHA），各视角视频在 channel 维度拼接后联合生成，保证多视角一致性。Given the current frame x and a text action a, the video model fVM(x, a) synthesizes a short-horizon video x1:S, predicting the sequence of visual outcomes after that action is executed. Long-horizon plans are extended to hundreds of frames by autoregressive chaining (recursively taking the last frame as the new starting point). On the multi-camera bimanual platform (ALOHA), the videos of all viewpoints are concatenated along the channel dimension and generated jointly, which guarantees cross-view consistency.
初始化 B 条平行规划 beam。每一步：① πVLM 为当前帧生成 A 个动作；② fVM 对每个动作合成 D 个视频分支（共 A×D 个）；③ HVLM 打分，选最高分视频加入当前 beam；④ 每 5 步，用最高分 beam 替换最低分 beam。最终输出 HVLM 得分最高的 beam 对应的视频计划。Initialize B parallel planning beams. At each step: ① πVLM generates A actions for the current frame; ② fVM synthesizes D video branches per action (A×D in total); ③ HVLM scores them and the highest-scoring video is appended to the current beam; ④ every 5 steps, the lowest-scoring beam is replaced by the highest-scoring one. The final output is the video plan of the beam with the highest HVLM score.
图 3：规划预算 vs. 成功率（左）与 7DoF 移动机械臂规划树（右）。 左图显示随 beam / branching factor 增大，"make line"任务成功率从 4% 升至 56%；右图展示规划树中 VLP 如何剪枝不合理动作分支（如未打开抽屉直接放置水果），保留正确的动作序列。Figure 3: Planning budget vs. success rate (left) and the planning tree of the 7DoF mobile manipulator (right). The left plot shows that as the beam / branching factor grows, the success rate on "make line" rises from 4% to 56%; the right panel shows how VLP prunes implausible action branches in the planning tree (for example placing fruit without first opening the drawer) and keeps the correct action sequence.
Goal-Conditioned Policy 执行视频计划Goal-Conditioned Policy Executes the Video Plan
给定合成视频计划 x1:H，goal-conditioned policy πcontrol(x, xg) 以当前帧 x 和下一目标帧 xg 为输入，输出低层控制动作 u，每帧执行固定步数。采用 receding horizon control（滚动时域控制）：执行固定步数后，用最新观测重新规划（replanning），以消除累积误差。Given the synthesized video plan x1:H, the goal-conditioned policy πcontrol(x, xg) takes the current frame x and the next goal frame xg as input and outputs low-level control actions u, executing a fixed number of steps per frame. Receding horizon control is adopted: after a fixed number of steps, the latest observation is used for replanning, which eliminates accumulated error.
03实验Experiments
实验在 Language Table 仿真环境和对应真实机器人（桌面机械臂）、7DoF 移动机械臂、14DoF 双臂 ALOHA 三个平台上进行。评估分两部分：（i）视频计划合成质量（人工判断视频是否完成任务，各方法各生成 50 条）；（ii）实际执行成功率（reward + completion rate）。Experiments are conducted on three platforms: the Language Table simulation environment and its corresponding real robot (a tabletop manipulator), a 7DoF mobile manipulator, and the 14DoF bimanual ALOHA. Evaluation has two parts: (i) video plan synthesis quality (humans judge whether the video completes the task; 50 plans are generated per method) and (ii) real execution success (reward + completion rate).
视频计划合成质量（Table 1）Video Plan Synthesis Quality (Table 1)
图 4：多视角视频计划（14DoF 双臂 ALOHA）。 VLP 在 4 个相机视角上同步生成视频计划，用于"将桌上所有物品堆叠"任务——包括堆叠碗、杯、餐具等一系列子步骤。VLP 仅给定第一帧图像，自动生成跨视角一致的长时视频。Figure 4: Multi-view video plans (14DoF bimanual ALOHA). VLP synthesizes video plans simultaneously across 4 camera views for the task "stack all objects on the table" — a series of sub-steps including stacking bowls, cups and utensils. Given only the first frame, VLP automatically generates long-horizon videos that stay consistent across views.
消融实验Ablations
规划预算（Table 3）：增加 beam 数（1→2）、规划时域（1→2）、branching factor（4→16），"make line" 任务完成率从 0% 上升至 16%，reward 从 48.9 升至 65.0。搜索力度越大，执行成功率持续提升。Planning budget (Table 3): increasing the number of beams (1→2), the planning horizon (1→2) and the branching factor (4→16) raises the "make line" completion rate from 0% to 16% and the reward from 48.9 to 65.0. The stronger the search, the higher the execution success rate.
动作提取方式（Table 4）：对比 inverse dynamics / goal policy (last frame) / goal policy (every frame) 三种从视频提取动作的方式。"goal-conditioned policy on every frame"在 Group-by-Color 任务上获得最高 reward（95.8）和最高完成率（92%），超过只用最后一帧的 85.0/66%，说明密集帧级控制更有效。Action extraction (Table 4): three ways of extracting actions from video are compared — inverse dynamics / goal policy (last frame) / goal policy (every frame). "goal-conditioned policy on every frame" attains the highest reward (95.8) and the highest completion rate (92%) on Group-by-Color, surpassing the last-frame-only variant at 85.0/66%, which shows that dense frame-level control is more effective.
图 5：仿真（左）与真实机器人（右）执行展示。 VLP 在仿真 Language Table 环境中完成"将所有方块移至左下角"、"按颜色分组"、"排成一横行"三类长时序任务；在真实机器人上同样成功执行相同任务。Figure 5: Execution in simulation (left) and on the real robot (right). In the simulated Language Table environment VLP completes three kinds of long-horizon tasks — "move all blocks to the bottom left", "group by color" and "make a line" — and it also executes the same tasks successfully on the real robot.
泛化能力Generalization
当 VLM 和 video model 在大规模 Internet 数据（含 YouTube 视频）上联合预训练后，VLP 能够泛化到：（i）训练集中未见过的新物体（如橡皮圈、纸杯蛋糕、木质六边形）；（ii）不同光照条件下的新环境；（iii）新任务指令（如"Pick snicker energy bar"、"Move moose toy near green pear"）。这种泛化能力来自将视频合成与低层控制解耦——video model 负责视觉动力学泛化，goal-conditioned policy 只需泛化到邻近视觉目标。After the VLM and the video model are jointly pretrained on large-scale Internet data (including YouTube videos), VLP generalizes to: (i) novel objects unseen in the training set (such as rubber bands, cupcakes and a wooden hexagon); (ii) new environments under different lighting conditions; (iii) new task instructions (such as "Pick snicker energy bar" and "Move moose toy near green pear"). This generalization comes from decoupling video synthesis from low-level control — the video model handles generalization of visual dynamics, while the goal-conditioned policy only needs to generalize to nearby visual goals.
04局限性Limitations
说明：以下局限性均为论文作者在 Section 5 "Limitations and Conclusion" 中明确陈述（stated），无推断内容。Note: All the limitations below are explicitly stated by the paper's authors in Section 5 "Limitations and Conclusion"; nothing is inferred.
图像表征无法捕捉完整三维状态Image representations cannot capture the full 3D state
"Our planning approach leverages images as a world state representation. In many tasks, this is insufficient as it does not capture the full 3D state and cannot encode latent factors such as physics or mass." 论文提出的缓解方向：生成多视角视频，或让 heuristic function 以完整视频为输入。"Our planning approach leverages images as a world state representation. In many tasks, this is insufficient as it does not capture the full 3D state and cannot encode latent factors such as physics or mass." Mitigations proposed in the paper: generate multi-view videos, or let the heuristic function take the full video as input.
视频动力学模型存在物理幻觉The video dynamics model hallucinates physics
"we observed that our video dynamics model does not always simulate dynamics accurately. In several situations, we observed that synthesized videos would make objects spontaneously appear or teleport to new locations."（物体凭空出现或瞬移）论文建议使用更大的 video model、更多训练数据、或引入显式强化学习反馈（如 RLHF for physics）来缓解。"we observed that our video dynamics model does not always simulate dynamics accurately. In several situations, we observed that synthesized videos would make objects spontaneously appear or teleport to new locations." (objects appearing out of nowhere or teleporting) The paper suggests mitigating this with a larger video model, more training data, or explicit reinforcement-learning feedback (such as RLHF for physics).
推理计算开销随搜索预算线性增长Inference compute grows linearly with the search budget
VLP 的计划质量随 beam 数和 branching factor 增大而提升，但推理时间也相应增长。在实时机器人控制场景中，需要在计划质量与响应延迟之间做出权衡。（作者在论文中提及此 scaling 特性，但未给出具体推理时延数据，此条为设计层面的隐含局限。）The plan quality of VLP improves as the number of beams and the branching factor grow, but inference time grows accordingly. In real-time robot control settings, a trade-off must be made between plan quality and response latency. (The authors mention this scaling property in the paper but give no concrete inference-latency figures; this entry is an implicit limitation at the design level.)
视频模型训练依赖带语言标注的短时轨迹数据Video model training depends on short-horizon trajectory data with language annotations
VLP 的 text-to-video model 需要短时图像轨迹片段与对应语言标签进行监督训练，数据采集与标注成本较高。尽管作者指出 VLP 可从不完整语言标注数据中受益（未标注段仍可用于学习动力学），但对新机器人平台和新任务域的数据需求仍构成实际部署的障碍。The text-to-video model of VLP requires supervised training on short-horizon image trajectory segments paired with language labels, and the cost of collecting and annotating such data is high. Although the authors point out that VLP can benefit from incompletely annotated language data (unlabeled segments can still be used to learn dynamics), the data requirements for new robot platforms and new task domains still constitute an obstacle to practical deployment.
