Learning Universal Policies via Text-Guided Video Generation
将文字描述转化为视频规划，实现跨环境通用策略Turning text descriptions into video plans for a universal cross-environment policy
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, Pieter Abbeel · MIT · Google · UC Berkeley
UniPi 将序列决策问题重新表述为文本条件视频生成问题：给定文本目标，扩散模型合成未来视频帧作为行动规划，随后通过 inverse dynamics 模型从视频中提取具体控制动作。利用图像作为跨环境统一表示，UniPi 实现了跨任务泛化、预训练迁移和真实机器人部署。UniPi reformulates sequential decision making as a text-conditioned video generation problem: given a text goal, a diffusion model synthesizes future video frames as an action plan, and an inverse dynamics model then extracts concrete control actions from the video. By using images as a unified cross-environment representation, UniPi achieves cross-task generalization, pretraining transfer, and real-robot deployment.
人工智能的核心目标之一是构建能够解决多样化任务的通用智能体。然而，传统强化学习方法面临两大根本挑战：其一，不同环境往往有各自独立的状态空间与动作空间，难以统一建模；其二，奖励函数的设计依赖于任务特定知识，无法跨环境迁移。与此同时，文本引导图像生成模型（如 DALL-E、Stable Diffusion）已展现出惊人的跨域组合泛化能力——仅凭文本描述即可生成从未见过的新图像。A central goal of artificial intelligence is to build general agents capable of solving diverse tasks. Conventional reinforcement learning, however, faces two fundamental challenges: first, different environments typically have their own state and action spaces, which are hard to model in a unified way; second, reward function design relies on task-specific knowledge and does not transfer across environments. Meanwhile, text-guided image generation models (such as DALL-E and Stable Diffusion) have shown striking combinatorial generalization across domains — generating never-before-seen images from a text description alone.
"A goal of artificial intelligence is to construct an agent that can solve a wide variety of tasks. Recent progress in text-guided image synthesis has yielded models with an impressive ability to generate complex novel images, exhibiting combinatorial generalization across domains. Motivated by this success, we investigate whether such tools can be used to construct more general-purpose agents."
核心洞察是：图像是跨环境的统一表示。无论是 Atari 游戏、机器人抓取还是导航任务，环境状态均可用图像帧表示，从而在统一的视觉空间中学习跨任务的规划能力。文本则作为目标规范，天然具备组合泛化性——不同语言短语的新组合即可指定新目标，无需重新训练。The key insight is that images are a unified representation across environments. Whether the task is an Atari game, robot grasping, or navigation, environment states can all be expressed as image frames, so planning ability can be learned across tasks in one shared visual space. Text then serves as the goal specification and is inherently combinatorially generalizable — new combinations of language phrases specify new goals with no retraining.
图1（来自论文 Figure 1）：UniPi 的核心思路。左：文本条件视频生成将规划表示为图像序列，inverse dynamics 模型从相邻帧差分中提取动作。右：UniPi 与传统 Transformer BC 的对比——UniPi 采用统一图像空间，可同时支持 Pick/Place 和 Relation 等多种任务类型，且在 Novel 场景下泛化能力更强。Figure 1 (from the paper's Figure 1): the core idea of UniPi. Left: text-conditioned video generation represents a plan as an image sequence, and the inverse dynamics model extracts actions from the difference between adjacent frames. Right: UniPi versus a conventional Transformer BC — UniPi works in a unified image space, can handle task types such as Pick/Place and Relation at the same time, and generalizes better in Novel settings.
72.6%预训练后在机器人抓取任务上的成功率Success rate on robot grasping tasks after pretraining
244.66FID↓（UniPi 视频质量，使用互联网预训练数据）FID↓ (UniPi video quality, using internet pretraining data)
8种不同机器人操作任务，统一策略同时处理distinct robot manipulation tasks, handled simultaneously by one unified policy
02方法 MethodMethod
UniPi（Universal Policy via video generation）由两个核心模块构成：①一个文本条件的视频扩散模型，作为轨迹规划器合成未来视频帧；②一个inverse dynamics 模型，从规划视频中回归具体控制动作。整个框架将视频生成与动作推断解耦，使得规划器可以共享跨环境的视觉知识。UniPi (Universal Policy via video generation) consists of two core modules: (i) a text-conditioned video diffusion model that acts as a trajectory planner and synthesizes future video frames; (ii) an inverse dynamics model that regresses concrete control actions from the planned video. The framework decouples video generation from action inference, so the planner can share visual knowledge across environments.
图2（来自论文 Figure 2 & Section 3）：UniPi 系统架构。上半部分为 Universal Policy（UniPi）：输入当前帧和文本目标，视频扩散模型（Trajectory Consistency Tiling）合成 H 步未来帧序列；下半部分为 Inverse Dynamics：输入合成视频的相邻帧对，预测每步动作，执行到真实环境中。Figure 2 (from the paper's Figure 2 & Section 3): the UniPi system architecture. Top, the Universal Policy (UniPi): given the current frame and a text goal, the video diffusion model (Trajectory Consistency Tiling) synthesizes an H-step sequence of future frames. Bottom, Inverse Dynamics: given pairs of adjacent frames from the synthesized video, it predicts the action at each step and executes it in the real environment.
Unified Decision Process (UPDP)
论文将标准 MDP 推广为 UPDP（Unified Decision Process），用六元组 G = (X, C, H, p) 定义：X 为观测空间（图像序列），C 为文本目标空间，H 为有限步数。规划器学习条件分布 π(·|{x_k}_{k=0}^{H}, c)，即给定初始帧和文本目标，生成 H 步图像轨迹；动作生成器 g(·|{x_k}_{k=0}^{H}) 再从视频中推断控制动作序列。这种分离允许：(1) 规划器跨环境共享视觉知识；(2) 动作推断独立适配不同机器人形态。The paper generalizes the standard MDP into a UPDP (Unified Decision Process), defined by the six-tuple G = (X, C, H, p): X is the observation space (image sequences), C the space of text goals, and H the finite horizon. The planner learns the conditional distribution π(·|{x_k}_{k=0}^{H}, c), i.e. given an initial frame and a text goal it produces an H-step image trajectory; the action generator g(·|{x_k}_{k=0}^{H}) then infers the control action sequence from the video. This separation allows: (1) the planner to share visual knowledge across environments; (2) action inference to be adapted independently to different robot embodiments.
Conditional Video Synthesis with Trajectory Consistency Tiling
视频扩散模型基于连续时间扩散框架，前向过程 q(x_t | x_0) = N(x_t; α_t x_0, σ_t² I)，反向过程通过去噪网络 s_θ(x_t, z_k, c) 从噪声恢复视频帧序列（其中 z_k 为初始帧条件，c 为文本 embedding）。关键技术创新是 Trajectory Consistency Tiling：将观测帧在时间维度上 tile 作为上下文，拼接到低分辨率条件视频中，再由超分辨率扩散网络细化；训练时随机混合有/无 tiling 两种模式（mixing probability），推理时强制使用 tiling 来保证环境状态一致性。作者也验证了 Deterministic、Stochastic（VAE 隐变量）等世界模型变体。The video diffusion model builds on a continuous-time diffusion framework. The forward process is q(x_t | x_0) = N(x_t; α_t x_0, σ_t² I), and the reverse process recovers the video frame sequence from noise through a denoising network s_θ(x_t, z_k, c) (where z_k is the initial-frame condition and c the text embedding). The key technical innovation is Trajectory Consistency Tiling: the observed frame is tiled along the temporal dimension as context and concatenated into the low-resolution conditioning video, which a super-resolution diffusion network then refines. Training randomly mixes the with-tiling and without-tiling modes (mixing probability), while inference always uses tiling to keep the environment state consistent. The authors also validate world-model variants such as Deterministic and Stochastic (VAE latent variables).
Inverse Dynamics for Action Extraction
Inverse dynamics 模型 g(a | x_k, x_{k+1}) 将相邻帧对作为输入，回归控制动作。在连续控制任务（机器人）中输出连续动作向量；在离散任务（Atari）中输出 softmax 分类。该设计不依赖奖励函数，可从视频演示中监督学习，与任何规划视频生成器配合使用。The inverse dynamics model g(a | x_k, x_{k+1}) takes a pair of adjacent frames as input and regresses the control action. For continuous-control tasks (robots) it outputs a continuous action vector; for discrete tasks (Atari) it outputs a softmax classification. The design does not rely on a reward function, can be supervised from video demonstrations, and works with any video plan generator.
03实验 ExperimentsExperiments
实验在三类场景中评估 UniPi：①语言条件机器人操作（CLIPort benchmark）；②多任务迁移（8 种机器人任务）；③真实世界机器人（Bridge dataset 上的预训练迁移）。评估指标包括任务完成率、CLIP score、FID、FVD。UniPi is evaluated in three kinds of settings: (i) language-conditioned robot manipulation (CLIPort benchmark); (ii) multi-task transfer (8 robot tasks); (iii) real-world robots (pretraining transfer on the Bridge dataset). Evaluation metrics include task completion rate, CLIP score, FID, and FVD.
Table 1（原文 Table 1）：Task Completion Accuracy in Combinatorial Environments。UniPi 在 novel 组合指令的 Relation 任务上以 74.4% 大幅超越所有 baseline（最佳竞争者 55.6%）。Table 1 (paper Table 1): Task Completion Accuracy in Combinatorial Environments. On Relation tasks with novel combinatorial instructions, UniPi reaches 74.4% and far surpasses every baseline (best competitor 55.6%).
图3（来自论文 Figure 3 & Figure 4）：UniPi 生成的合成视频示例（左列为文本指令，中列为合成视频帧，右列为 inverse dynamics 执行后的真实状态）以及 Action Execution 流程图——视频帧序列通过 inverse dynamics 模型映射为机器人动作序列，在真实环境中执行。Figure 3 (from the paper's Figure 3 & Figure 4): examples of synthetic video generated by UniPi (left column, the text instruction; middle column, the synthesized video frames; right column, the real states after inverse dynamics execution), together with the Action Execution pipeline — the video frame sequence is mapped by the inverse dynamics model into a robot action sequence and executed in the real environment.
Table 4（原文 Table 4）：在 Bridge dataset 预训练后，FID 从 288.02 降至 244.66，视频质量显著提升（78.1% 的成功率提升见原文详细消融结果）。Table 4 (paper Table 4): after pretraining on the Bridge dataset, FID drops from 288.02 to 244.66, a marked gain in video quality (the 78.1% success-rate gain is reported in the paper's detailed ablation results).
图4（来自论文 Figure 8 & Figure 9）：左：High Fidelity Plan Generation——预训练使 UniPi 能根据语言指令生成高质量视频规划（含颜色/空间组合泛化）。右：Robustness to Background Change——UniPi 对黑色裁剪背景、添加照片遮挡等扰动具有较强鲁棒性，从头训练的模型则经常失败。Figure 4 (from the paper's Figure 8 & Figure 9): Left, High Fidelity Plan Generation — pretraining lets UniPi generate high-quality video plans from a language instruction (including color/spatial combinatorial generalization). Right, Robustness to Background Change — UniPi is fairly robust to perturbations such as a black cropped background or an added photo occlusion, whereas a model trained from scratch often fails.
消融实验Ablations
消融实验（原文 Table 2）验证了各组件的必要性：Ablations (paper Table 2) confirm that each component is necessary:
Condition
Hierarchy
Temporally Consistent
Place
Relation
Yes
Yes
Yes
52.2 ± 2.2
34.5 ± 2.1
Yes
No
No
51.1 ± 5.6
34.4 ± 2.7
Yes
Yes
No
48.9 ± 2.8
31.1 ± 2.6
Yes（Full UniPi）
Yes
Yes
79.4 ± 4.5
74.4 ± 2.5
Table 2（原文 Table 2）：Condition（文本条件）、Hierarchy（超分辨率层次结构）、Temporally Consistent（Tiling 一致性）三者缺一不可；完整 UniPi 在 novel 组合指令上比任意消融版本高出约 40 个百分点。Table 2 (paper Table 2): Condition (text conditioning), Hierarchy (super-resolution hierarchy) and Temporally Consistent (Tiling consistency) are all indispensable; on novel combinatorial instructions the full UniPi is about 40 percentage points above any ablated version.
04局限性 LimitationsLimitations
说明：以下局限性部分为论文正文明确陈述（labeled "stated"），部分为从设计中推断（labeled "inferred"）。Note: Some of the limitations below are explicitly stated in the paper text (labeled "stated"); others are inferred from its design (labeled "inferred").
规划速度慢（stated）Slow planning (stated)
扩散模型的迭代去噪过程较慢，视频规划的生成时间显著长于直接策略前向推断。论文中未给出具体延迟数字，但指出这是实时部署的主要瓶颈。加速采样（如 DDIM、一步蒸馏）是未来方向。The iterative denoising of a diffusion model is slow, so generating a video plan takes markedly longer than a direct policy forward pass. The paper gives no concrete latency figures, but points out that this is the main bottleneck for real-time deployment. Faster sampling (e.g. DDIM or one-step distillation) is a future direction.
依赖 inverse dynamics 精度（inferred）Dependence on inverse dynamics accuracy (inferred)
动作提取模块假设相邻帧差分可以准确还原控制信号。若视频规划中存在跳帧、模糊或幻觉帧，inverse dynamics 可能产生错误动作。论文未系统分析视频幻觉对任务成功率的影响。The action extraction module assumes that the difference between adjacent frames faithfully recovers the control signal. If the video plan contains skipped, blurred, or hallucinated frames, inverse dynamics may produce wrong actions. The paper does not systematically analyze how video hallucination affects task success rate.
视频幻觉与非物理轨迹（inferred）Video hallucination and non-physical trajectories (inferred)
生成式视频模型有时会产生物理上不可能的帧序列（如物体穿透、瞬移），这类幻觉轨迹无法被 inverse dynamics 正确执行。真实机器人上的鲁棒性实验（Figure 9）显示对背景扰动有较好鲁棒性，但对幻觉的系统性研究仍缺乏。Generative video models sometimes produce physically impossible frame sequences (objects passing through each other, teleporting), and such hallucinated trajectories cannot be executed correctly by inverse dynamics. The real-robot robustness experiment (Figure 9) shows good robustness to background perturbation, but a systematic study of hallucination is still lacking.
论文中实验以短视界 episodic 任务为主（H 步固定）。对需要数百步的长视界任务，视频扩散模型的一致性和计划可行性尚未充分验证。作者指出层次化规划是解决长视界问题的未来方向。The experiments are dominated by short-horizon episodic tasks (with H steps fixed). For long-horizon tasks requiring hundreds of steps, the consistency and plan feasibility of video diffusion models have not been sufficiently validated. The authors point to hierarchical planning as the future direction for long-horizon problems.
