Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow

01 动机Motivation

前沿视频生成模型已能合成逼真的人类操作场景，但如何将其中蕴含的物理知识转化为机器人可执行的指令，仍是一个开放难题。核心挑战在于：视频描述的是物体状态变化，而机器人需要的是驱动这些变化的执行器动作——两者之间存在巨大的 embodiment gap。Frontier video generation models can already synthesize realistic scenes of human manipulation, but how to turn the physical knowledge they embody into commands a robot can execute remains an open problem. The core challenge is that a video describes changes of object state, whereas a robot needs the actuator motions that drive those changes — a huge embodiment gap lies between the two.

"Generative video modeling has emerged as a compelling tool to zero-shot reason about plausible physical interactions for open-world manipulation. Yet, it remains a challenge to translate such human-led motions into the low-level actions demanded by robotic systems."

现有方法通常试图直接从视频帧预测关节角度，但视频模型对具身细节（手型、关节构型）的模拟往往失真，导致难以直接跟随。Dream2Flow 的洞察是：物体的运动轨迹比执行器动作更稳定、更可迁移——无论使用哪种机器人，"把面包放进碗里"这一动作都要求面包沿相似的 3D 路径运动。Existing methods usually try to predict joint angles directly from video frames, but video models often distort embodiment details (hand shape, joint configuration), which makes them hard to follow directly. The insight of Dream2Flow is that the motion trajectory of the object is more stable and more transferable than actuator motion: whichever robot is used, "putting the bread into the bowl" requires the bread to move along a similar 3D path.

Paper teaser showing the Dream2Flow framework overview — **图 1（论文 Figure 1）：Dream2Flow 总览。** 系统接受任务指令（如 "Open the toaster oven"）和 RGB-D 初始图像，由视频生成模型合成操作视频，再从中提取 3D object flow，最终指导机器人完成推、放、开、盖、扫、回收等多种真实任务。整个过程无需任务专属演示数据。**Figure 1 (paper Figure 1): overview of Dream2Flow.** The system takes a task instruction (e.g. "Open the toaster oven") and an initial RGB-D image, synthesizes a manipulation video with a video generation model, extracts 3D object flow from it, and finally guides the robot through diverse real tasks such as pushing, placing, opening, covering, sweeping and recycling. The whole process needs no task-specific demonstration data.

60真实世界总评估试验次数Real-world evaluation trials in total

80%端到端任务成功率（40/44 成功流）End-to-end task success rate (40/44 successful flows)

4支持物体类别：刚性 / 关节 / 可变形 / 颗粒Object categories supported: rigid / articulated / deformable / granular

3测试机器人平台（Franka、Spot、GR1）Robot platforms tested (Franka, Spot, GR1)

02 方法Method

Dream2Flow 是一条三阶段 pipeline：视频生成 → 3D flow 提取 → 机器人控制。关键设计是以 3D object flow 解耦"物体做了什么"与"执行器如何驱动"，使跨具身迁移成为可能。Dream2Flow is a three-stage pipeline: video generation → 3D flow extraction → robot control. The key design uses 3D object flow to decouple "what the object did" from "how the actuator drives it", making cross-embodiment transfer possible.

Dream2Flow method pipeline diagram — **图 2（论文 Figure 2）：方法概览。** 给定自然语言指令和 RGB-D 观测，视频生成模型生成操作视频帧；视觉基础模型从中提取物体 mask、视频深度和 2D 点轨迹，重建 3D object flow；最终机器人策略跟踪该 3D flow 产生低层控制指令。**Figure 2 (paper Figure 2): method overview.** Given a natural-language instruction and an RGB-D observation, the video generation model produces manipulation video frames; vision foundation models extract the object mask, video depth and 2D point tracks from them and reconstruct the 3D object flow; finally the robot policy tracks that 3D flow to produce low-level control commands.

Video Generation（视频生成）Video Generation

以初始 RGB-D 图像和任务指令为条件，调用 image-to-video 生成模型合成任务执行视频帧序列。系统采用 Veo 3 [90] 等开箱即用的视频生成器，无需针对机器人场景微调。Conditioned on the initial RGB-D image and the task instruction, an image-to-video generation model is called to synthesize the video frame sequence of the task being executed. The system uses off-the-shelf video generators such as Veo 3 [90], with no fine-tuning for robotic scenes.

3D Object Flow Extraction（三维流提取）3D Object Flow Extraction

利用视觉基础模型串联完成三步：① 用初始深度图初始化物体 mask；② 用 SpatialTrackerV2 [80,81] 从视频帧估计每帧的视频深度；③ 用 CoTracker [1] 追踪 2D 点轨迹，结合深度反投影得到以初始帧为参考的 3D 轨迹 P_1:T。该表示从物体掩码内均匀采样 n 个点，以初始深度 D₀ 对第一帧对齐，输出物体中心 3D flow。Vision foundation models are chained in three steps: ① initialize the object mask with the initial depth map; ② estimate the per-frame video depth from the video frames with SpatialTrackerV2 [80,81]; ③ track 2D point trajectories with CoTracker [1] and back-project them with depth to obtain the 3D trajectory P_1:T referenced to the initial frame. The representation samples n points uniformly inside the object mask, aligns the first frame with the initial depth D₀, and outputs an object-centric 3D flow.

Action Inference via 3D Object Flow（动作推断）Action Inference via 3D Object Flow

根据物体类型和机器人平台，采用两种策略将 3D flow 转为低层指令：
• Trajectory Optimization（轨迹优化）：适用于刚性物体（如 AVDC [2]），以末端执行器位姿序列跟踪重建的 3D 物体轨迹，并加入鼓励匹配 3D flow 同时惩罚碰撞的奖励。
• Reinforcement Learning（强化学习）：适用于可变形、颗粒状等难以直接做轨迹优化的物体（如 RIGVID [56]），以 3D object flow 作为奖励信号训练无演示 RL 策略。Depending on the object type and the robot platform, two strategies turn 3D flow into low-level commands:
• Trajectory Optimization: for rigid objects (as in AVDC [2]), an end-effector pose sequence tracks the reconstructed 3D object trajectory, with a reward that encourages matching the 3D flow while penalizing collisions.
• Reinforcement Learning: for deformable, granular and other objects on which trajectory optimization is hard to apply directly (as in RIGVID [56]), the 3D object flow serves as the reward signal for training a demonstration-free RL policy.

为何选择 3D Object Flow 作为中间表示？Why choose 3D object flow as the intermediate representation?

论文指出，3D flow 具备三项关键优势：（1）跨具身可迁移——物体运动与机器人关节空间解耦；（2）适配多种物体类型——刚性、关节式、可变形、颗粒状均适用；（3）无需任务演示——直接从视频生成模型的输出中提取，零样本使用预训练能力。The paper points out three key advantages of 3D flow: (1) cross-embodiment transferability — object motion is decoupled from the robot joint space; (2) coverage of diverse object types — rigid, articulated, deformable and granular objects all apply; (3) no task demonstrations required — it is extracted directly from the output of the video generation model, using pre-trained capability zero-shot.

"By separating the state changes from the actuators that realize those changes, Dream2Flow overcomes the embodiment gap and enables zero-shot guidance from pre-trained video models."

03 实验Experiments

实验在仿真（Push-T 任务）和真实世界（60 次试验，含推、放、开、盖、扫、回收六类任务）中评估 Dream2Flow，测试对象涵盖 Franka Panda、Boston Dynamics Spot、Fourier GR1 三种机器人。Dream2Flow is evaluated in simulation (the Push-T task) and in the real world (60 trials covering six task types: pushing, placing, opening, covering, sweeping and recycling), with three robots under test: Franka Panda, Boston Dynamics Spot and Fourier GR1.

真实世界任务成功率（Table I，论文）Real-world task success rate (paper Table I)

以下为真实机器人 60 次试验的各阶段通过率：Per-stage pass rates over the 60 real-robot trials:

阶段Stage	尝试次数Attempts	成功次数Successes	成功率Success rate
Video Generation（视频生成）Video Generation	60	48	80%
Flow Extraction（流提取）Flow Extraction	48	44	92%
Robot Execution（机器人执行）Robot Execution	44	40	91%

Robustness evaluation and qualitative task results — **图 3（论文 Figure 3 + 4）：任务初始状态与鲁棒性评估。** 左侧展示 Push-T、Put Bread in Bowl、Open Oven、Cover Bowl、Open Door 五类任务的初始与目标状态；右侧为鲁棒性柱状图：在相机视角（Camera Obl.）、物体实例（Object Instances）、背景（Backgrounds）和视角（Viewing Angles）的变化下，Dream2Flow 与参考设置相比下降不超过 20%，表现稳定。**Figure 3 (paper Figure 3 + 4): task initial states and robustness evaluation.** Left: the initial and goal states of five task types — Push-T, Put Bread in Bowl, Open Oven, Cover Bowl and Open Door; right: the robustness bar chart — under variations of camera obliquity (Camera Obl.), object instances (Object Instances), backgrounds (Backgrounds) and viewing angles (Viewing Angles), Dream2Flow drops by no more than 20% relative to the reference setting, remaining stable.

仿真对比（Push-T，Table III）Simulation comparison (Push-T, Table III)

在 Push-T 任务上，Dream2Flow 使用 Veo 3 视频生成器（Veo 3 [90]），与 Wan2.1 [99] 和 Kling 2.1 等视频模型对比：On the Push-T task, Dream2Flow uses the Veo 3 video generator (Veo 3 [90]) and is compared with video models such as Wan2.1 [99] and Kling 2.1:

视频模型Video model	Push-T 成功率Push-T success rate	Open Oven 成功率Open Oven success rate
Wan2.1 [99]	32/100	2/10
Kling 2.1	10/100	4/10
Veo 3 [90]	82/100	8/10

Results tables: real-world manipulation and cross-embodiment RL reward comparison — **图 4（论文 Table I + II）：真实操控与跨具身 RL 奖励对比。** 左表显示真实机器人各类任务（Bread in Bowl、Open Oven、Cover Bowl）中 Dream2Flow 相对 AVDC [2] 和 RIGVID [56] 的成功率提升；右表（Table II）对比不同奖励类型（Franka / Spot / GR1）下 Object State（环境状态）奖励与 3D Object Flow 奖励的最终策略性能，后者在 Franka 和 Spot 平台达到 100/100。**Figure 4 (paper Table I + II): real-world manipulation and cross-embodiment RL reward comparison.** The left table shows the success-rate gains of Dream2Flow over AVDC [2] and RIGVID [56] on the real-robot tasks (Bread in Bowl, Open Oven, Cover Bowl); the right table (Table II) compares the final policy performance of the Object State reward against the 3D Object Flow reward under different reward types (Franka / Spot / GR1), the latter reaching 100/100 on the Franka and Spot platforms.

失效分析与消融Failure analysis and ablation

Failure breakdown and dynamics model ablation — **图 5（论文 Figure 7 + Table IV）：失效分析与粒子动力学模型消融。** 上图（Figure 7）将 60 次真实试验失效分解为：Object Morphing（物体形态突变，6 次）、Hallucination（幻觉，6 次）、Flow Extraction（流提取失败，4 次）、Robot Execution（机器人执行失败，4 次）。下表（Table IV）对比粒子动力学模型类型（Flow / Heuristic / Particle），结果显示 Particle 模型成功率最高（82/100），大幅领先 Heuristic（17/100）和 Flow（12/100），说明精准的粒子运动建模对颗粒类任务至关重要。**Figure 5 (paper Figure 7 + Table IV): failure analysis and particle dynamics model ablation.** The upper panel (Figure 7) breaks the failures of the 60 real trials down into Object Morphing (abrupt change of object shape, 6 cases), Hallucination (6 cases), Flow Extraction (extraction failure, 4 cases) and Robot Execution (execution failure, 4 cases). The lower table (Table IV) compares particle dynamics model types (Flow / Heuristic / Particle): the Particle model attains the highest success rate (82/100), far ahead of Heuristic (17/100) and Flow (12/100), showing that accurate particle motion modeling is essential for granular tasks.

论文还测试了 Dream2Flow 在同一场景多任务（multi-task in the same scene）中的表现（Figure 5）：给定相同的环境布局，仅更换语言指令，系统即可切换目标物体完成不同操作，验证了框架的通用性。The paper also tests Dream2Flow on multi-task in the same scene (Figure 5): given the same environment layout and only swapping the language instruction, the system can switch target objects and complete different manipulations, verifying the generality of the framework.

04 局限性Limitations

Note: 以下局限性部分来自论文 Section V（Conclusion）明确陈述，部分为从方法设计中推断（标注 inferred）。Some of the limitations below are stated explicitly in Section V (Conclusion) of the paper; others are inferred from the method design (marked inferred).

视频生成质量瓶颈Video generation quality bottleneck

论文指出，视频生成失败（object morphing 和 hallucination）占所有失效的 50%（12/24 次）。当视频模型生成形态不一致的物体（如形状骤变）或幻觉内容（如物理上不可能的运动），下游的 flow 提取和机器人执行均无法正常工作。系统性能严重依赖所使用视频生成器的质量，且不同模型差异显著（Push-T 成功率从 Kling 的 10% 到 Veo 3 的 82%）。The paper notes that video generation failures (object morphing and hallucination) account for 50% of all failures (12/24 cases). When the video model generates objects of inconsistent morphology (e.g. abrupt shape changes) or hallucinated content (e.g. physically impossible motion), neither the downstream flow extraction nor the robot execution can work properly. System performance depends heavily on the quality of the video generator used, and models differ markedly (Push-T success rate ranges from 10% for Kling to 82% for Veo 3).

遮挡与跟踪失败Occlusion and tracking failure

3D flow 提取依赖 CoTracker 和 SpatialTrackerV2 等视觉跟踪工具，当物体被严重遮挡或离开相机视野（论文提及"相机旋转导致点超出视野"）时，点轨迹丢失，导致 flow 提取失败（60 次试验中有 4 次）。此外，Cover Bowl 任务中物体常处于遮挡状态，使执行变得困难。3D flow extraction relies on visual tracking tools such as CoTracker and SpatialTrackerV2; when the object is heavily occluded or leaves the camera field of view (the paper mentions "camera rotation pushing points out of view"), point tracks are lost and flow extraction fails (4 of the 60 trials). In addition, objects in the Cover Bowl task are often occluded, which makes execution difficult.

刚性假设与物体类型限制（inferred）Rigidity assumption and object-type limits (inferred)

轨迹优化路径（AVDC-based）假设所跟踪物体为刚体，通过 6D 位姿变换描述运动。对于高度可变形物体（如布料）或拓扑结构复杂的颗粒状物体（如意大利面），系统切换至 RL 路径，但 RL 训练需在仿真器中进行，存在 sim-to-real gap。The trajectory-optimization path (AVDC-based) assumes the tracked object is a rigid body and describes its motion by a 6D pose transform. For highly deformable objects (e.g. cloth) or granular objects with complex topology (e.g. spaghetti), the system switches to the RL path, but RL training has to be carried out in a simulator, which leaves a sim-to-real gap.

依赖 RGB-D 输入与外部模型（inferred）Dependence on RGB-D input and external models (inferred)

系统初始化需要深度信息（D 通道），限制了在纯 RGB 场景中的部署。此外，整条 pipeline 串联了多个外部预训练模型（视频生成器、CoTracker、SpatialTrackerV2），任一模型更新或不可用均可能影响系统稳定性，推理延迟也因此较高。System initialization requires depth information (the D channel), which limits deployment in pure-RGB scenes. Moreover, the whole pipeline chains several external pre-trained models (the video generator, CoTracker, SpatialTrackerV2); an update to or unavailability of any one of them may affect system stability, and inference latency is correspondingly high.