Learning to Act from Actionless Videos through Dense Correspondences

01 动机 MotivationMotivation

机器人策略训练面临两大核心困难：其一，收集带动作标注的演示数据成本极高，且高度依赖特定平台；其二，仿真与真实环境的 sim-to-real gap 使得策略难以直接部署。互联网上已存在海量 RGB 视频，如果能直接从这些无动作标签的视频中学习策略，将大幅降低数据采集门槛并提升泛化能力。Training robot policies faces two core difficulties. First, collecting demonstration data with action annotations is extremely costly and highly platform-specific; second, the sim-to-real gap between simulation and the real world makes policies hard to deploy directly. Massive amounts of RGB video already exist on the internet, and being able to learn policies directly from such actionless videos would greatly lower the barrier to data collection and improve generalization.

"We present a method for training a robot policy capable of reliably executing diverse tasks across different robots and environments from RGB video demonstrations without any action annotation."

先前方法的局限性主要体现在两方面：
（1）基于逆动力学模型的方法（如 UniPi）需要知道精确步数，且通常以模态方式预测动作，训练开销大（需超过 256 TPU-pods）；
（2）行为克隆（BC）类方法依赖专家动作标注，无法直接利用互联网视频资源。
AVDC 的核心洞见是：图像本身即可同时编码状态与动作信息——相邻帧之间的稠密光流对应关系天然蕴含了物体运动轨迹，可以以闭合形式解算 SE(3) 变换，无需学习任何额外动作预测器。The limitations of prior methods lie mainly in two respects:
(1) Inverse-dynamics-model based methods (such as UniPi) need to know the exact number of steps, usually predict actions in a modal fashion, and are expensive to train (more than 256 TPU-pods required);
(2) Behavior cloning (BC) methods rely on expert action annotations and cannot directly exploit internet video resources.
The key insight of AVDC is that images themselves already encode both state and action information — the dense optical-flow correspondences between adjacent frames naturally carry object motion trajectories, from which the SE(3) transformation can be solved in closed form, without learning any extra action predictor.

02 方法 MethodMethod

AVDC 将策略执行解耦为三个阶段：（a）条件视频生成——用扩散模型合成从当前状态到目标的想象执行视频；（b）稠密光流估计——计算相邻帧的稠密对应关系；（c）动作回归——结合深度信息，以闭合形式将光流对应关系转化为 SE(3) 刚体变换，指导机器人执行。AVDC decouples policy execution into three stages: (a) conditioned video generation — a diffusion model synthesizes an imagined execution video from the current state to the goal; (b) dense optical flow estimation — dense correspondences between adjacent frames are computed; (c) action regression — combined with depth information, the flow correspondences are turned in closed form into SE(3) rigid transformations that drive the robot.

条件视频生成（Conditioned Video Generation）Conditioned Video Generation

本文使用以 U-Net（基于 Ho et al. 2019）为骨干的条件扩散模型，输入当前帧 f₀ 和文本目标描述，预测未来 F 帧序列。为提升时间一致性，在空间注意力中引入factorized spatial-temporal convolution（参考 Ho et al. 2022），使每帧在自身空间范围内同时与时间维度交互。推理时采用 DDIM 加速，仅需 10 步即可生成高保真视频（标准需 100 步）。The paper uses a conditional diffusion model with a U-Net backbone (based on Ho et al. 2019), taking the current frame f₀ and a text goal description as input and predicting a sequence of F future frames. To improve temporal consistency, factorized spatial-temporal convolution (following Ho et al. 2022) is introduced into the spatial attention, so that each frame interacts with the temporal dimension within its own spatial extent. At inference, DDIM acceleration yields high-fidelity videos in only 10 steps (the standard setting needs 100).

光流预测（Flow Prediction）Flow Prediction

给定合成视频中相邻帧对 (img_t, img_t+1)，利用 off-the-shelf 光流估计器（如 RAFT）提取稠密对应关系。每个像素的光流向量编码了该点在下一时刻的位移，从而构成从当前帧到下一帧的稠密像素级映射。Given a pair of adjacent frames (img_t, img_t+1) from the synthesized video, an off-the-shelf optical flow estimator (such as RAFT) extracts dense correspondences. The flow vector of each pixel encodes the displacement of that point at the next time step, thus forming a dense pixel-level mapping from the current frame to the next one.

动作回归：从光流到 SE(3)（Action Regression from Flows and Depths）Action Regression from Flows and Depths

这是 AVDC 的核心创新。给定深度图和稠密光流，可以将每个 3D 点的对应关系 (x_t, x̂_t+1) 提取出来，从而以闭合形式（closed-form）求解刚体变换 T = (R, t) ∈ SE(3)，最小化如下目标：This is the core innovation of AVDC. Given the depth map and the dense optical flow, the correspondence (x_t, x̂_t+1) of every 3D point can be extracted, so that the rigid transformation T = (R, t) ∈ SE(3) is solved in closed form by minimizing the following objective:

min_{R∈SO(3), t∈ℝ³} Σ_i ‖R·x_t,i + t − x̂_t+1,i‖²

对于操控任务（manipulation），系统先识别目标对象掩码，再仅在对象区域内计算变换，得到末端执行器的 pick-and-place 轨迹。对于导航任务（navigation），全帧光流被用来估计相机（即机器人底盘）的运动。整个动作推导无需学习任何参数，完全依赖几何约束，因此具有强大的跨平台迁移能力。For manipulation tasks, the system first identifies the mask of the target object and then computes the transformation only within the object region, obtaining the pick-and-place trajectory of the end effector. For navigation tasks, the full-frame optical flow is used to estimate the motion of the camera, i.e. of the robot base. The entire action derivation learns no parameters at all and depends purely on geometric constraints, which gives it strong cross-platform transferability.

Replanning 策略Replanning Strategy

在执行过程中，机器人每完成一个子目标后，以新的当前观测作为条件重新运行视频生成和动作回归流程（replanning），以应对执行误差或环境变化。实验表明随着 replanning 次数增加，任务成功率单调上升（图 5）。During execution, after the robot completes each subgoal it re-runs the video generation and action regression pipeline conditioned on the new current observation (replanning), so as to cope with execution error or environment change. Experiments show that the task success rate increases monotonically with the number of replanning rounds (Figure 5).

03 实验 ExperimentsExperiments

在三大基准上验证 AVDC：Meta-World（桌面操控仿真，11 任务，165 视频）、iTHOR（室内导航仿真，12 类目标，240 视频）、Bridge / Franka Panda（真实机器人，zero-shot 迁移）。基线包括行为克隆（BC-Scratch、BC-R3M）、UniPi 及 AVDC 各变体。AVDC is validated on three benchmarks: Meta-World (tabletop manipulation simulation, 11 tasks, 165 videos), iTHOR (indoor navigation simulation, 12 object categories, 240 videos) and Bridge / Franka Panda (real robots, zero-shot transfer). Baselines include behavior cloning (BC-Scratch, BC-R3M), UniPi and the AVDC variants.

Meta-World 操控结果Meta-World Manipulation Results

iTHOR 导航结果iTHOR Navigation Results

在 iTHOR 室内导航基准上，AVDC 在 12 类目标物体上均表现出合理的导航成功率。系统以当前 RGBD 帧和目标对象名称为输入，在合成视频中追踪全帧光流以估算摄像机运动，并循环执行 replanning 直到到达目标区域。On the iTHOR indoor navigation benchmark, AVDC attains reasonable navigation success rates on all 12 categories of target objects. Taking the current RGBD frame and the name of the target object as input, the system tracks the full-frame optical flow in the synthesized video to estimate camera motion, and keeps replanning in a loop until it reaches the target region.

跨实体迁移：Bridge 数据集 → Franka PandaCross-Embodiment Transfer: Bridge Dataset → Franka Panda

在此实验中，AVDC 模型在包含 ~200 段人类手部推物视频的 Bridge 数据集上训练，然后零样本迁移到 Franka Emika Panda 机械臂。由于系统在动作推导上完全依赖几何对应关系而非具体动作标注，因此无需任何域适应即可直接部署。定性实验表明机械臂可以可靠跟随合成视频中的物体运动轨迹（图 9-10）。In this experiment the AVDC model is trained on the Bridge dataset, which contains ~200 videos of a human hand pushing objects, and is then transferred zero-shot to a Franka Emika Panda arm. Because the action derivation depends entirely on geometric correspondences rather than on concrete action annotations, it can be deployed directly without any domain adaptation. Qualitative experiments show that the arm reliably follows the object motion trajectories in the synthesized video (Figures 9-10).

消融实验（Ablations）Ablations

• AVDC (Flow)：仅用光流预测动作，无 replanning——在 faucet-close 等简单任务表现尚可，但在需要精确接触的任务上（如 button-top）明显下降。AVDC (Flow): actions predicted from optical flow only, without replanning — acceptable on simple tasks such as faucet-close, but a clear drop on tasks that require precise contact (such as button-top).
• AVDC (No Replan)：完整管线但不做 replanning——验证单次规划的上限，成功率明显低于 Full 版。AVDC (No Replan): the full pipeline but without replanning — it probes the ceiling of single-shot planning, and its success rate is clearly below that of the Full version.
• AVDC (Full)：完整两阶段（生成+重规划）管线，性能最优，在 Meta-World 上大幅超过 BC 和 UniPi 基线。AVDC (Full): the complete two-stage (generation + replanning) pipeline, the best performing variant, far exceeding the BC and UniPi baselines on Meta-World.
• Replanning 次数分析（图 5 右图）：成功率随次数增加单调上升，在约 10 次后趋于饱和。Analysis of the number of replanning rounds (Figure 5, right): the success rate rises monotonically with more rounds and saturates after about 10.

04 局限性 LimitationsLimitations