EgoEngine: From Egocentric Human Videos to High-Fidelity Dexterous Robot Demonstrations

01 动机Motivation

灵巧操作的大规模数据采集代价高昂——遥操作依赖专业硬件、复杂接口与高自由度接触控制，难以扩展。相比之下，第一视角人类视频天然记录了多样场景中的接触丰富操作行为，是廉价且可扩展的监督来源。然而，将人类视频直接用于机器人学习面临两大障碍。Large-scale data collection for dexterous manipulation is prohibitively expensive: teleoperation relies on specialized hardware, complex interfaces, and high-degree-of-freedom contact control, and therefore scales poorly. Egocentric human videos, by contrast, naturally record contact-rich manipulation behaviors across diverse scenes and constitute a cheap, scalable source of supervision. Using human videos directly for robot learning, however, faces two major obstacles.

"Human videos are not robot demonstrations. The challenge is twofold: visually, human arms and hands occlude the scene and differ substantially from the robot embodiment; on the action side, differences in morphology, kinematics, actuation, and contact dynamics make directly retargeted robot trajectories physically infeasible."

EgoEngine 同时应对两个差距：视觉差距（visual gap）——人类手臂遮挡场景且外观与机器人截然不同；动作差距（action gap）——人类运动直接重定向给机器人后，因形态学、运动学与接触动力学不匹配而不可执行。作者提出以物体为中心的「视觉–动作双支生成管线」，将这两个差距的弥合统一在同一框架内。EgoEngine addresses both gaps at once: the visual gap - human arms occlude the scene and look entirely unlike the robot embodiment; and the action gap - human motion retargeted directly onto the robot is not executable because of mismatches in morphology, kinematics, and contact dynamics. The authors propose an object-centric visual-action dual-branch generation pipeline that unifies the bridging of both gaps within a single framework.

02 方法Method

给定第一视角 RGB 视频，EgoEngine 首先构建物体中心的数字孪生（相机几何、深度、6D 物体轨迹、手部与物体 mask），然后并行运行两条支路：动作支路将人类运动转化为可执行机器人动作；视觉支路将人类帧转化为机器人视角观测。两条支路联合输出成对的机器人示范 (õ_t, ã_t)。Given an egocentric RGB video, EgoEngine first builds an object-centric digital twin (camera geometry, depth, 6D object trajectories, hand and object masks), then runs two branches in parallel: the action branch converts human motion into executable robot actions, while the visual branch converts human frames into robot-view observations. Jointly the two branches output paired robot demonstrations (õ_t, ã_t).

数字孪生构建（Human Video to Simulation）Digital Twin Construction (Human Video to Simulation)

使用 Aria Gen2 眼镜采集同步 RGB 帧与每帧 21 个手部关键点的 3D 姿态。FoundationStereo 估计绝对深度图；SAM2 通过手部关键点提示生成人手 mask，通过首帧点提示追踪任务物体 mask；FoundationPose 在 RGBD 帧上估计时序一致的 6D 物体轨迹 {T^o_t}^T_t=1。上述相机几何、深度、mask、手部姿态、物体网格与物体轨迹共同构成数字孪生。Aria Gen2 glasses capture synchronized RGB frames together with the 3D poses of 21 hand keypoints per frame. FoundationStereo estimates absolute depth maps; SAM2 produces the human-hand mask from hand-keypoint prompts and tracks the task-object mask from first-frame point prompts; FoundationPose estimates temporally consistent 6D object trajectories {T^o_t}^T_t=1 on RGBD frames. The camera geometry, depth, masks, hand poses, object meshes, and object trajectories above jointly constitute the digital twin.

动作生成：以人类为中心的重定向（Human-Centric Retargeting）Action Generation: Human-Centric Retargeting

给定人类视频中 5 根手指指尖的位置与朝向 {(p^t_tip,k, R^t_tip,k)}⁵_k=1 及腕部朝向 R^t_wrist，使用 MINK 求解逆运动学（IK）：Given the positions and orientations of the 5 fingertips in the human video {(p^t_tip,k, R^t_tip,k)}⁵_k=1 and the wrist orientation R^t_wrist, inverse kinematics (IK) is solved with MINK:

q^*_t = arg min L_tip(q; t) + λ_w L_wrist(q; t)，subject to 关节限位与自碰撞约束。q^*_t = arg min L_tip(q; t) + λ_w L_wrist(q; t), subject to joint-limit and self-collision constraints.

得到参考轨迹 τ^ref = {q^*_t}^T_t=1，作为后续仿真优化的运动先验。This yields the reference trajectory τ^ref = {q^*_t}^T_t=1, which serves as the motion prior for the subsequent simulation-based optimization.

动作生成：MCTS-style 自适应模式切换（Object-Centric Trajectory Optimization）Action Generation: MCTS-style Adaptive Mode Switching (Object-Centric Trajectory Optimization)

重定向轨迹在形态学与接触动力学不匹配下往往不可执行。EgoEngine 在仿真中以物体中心目标对轨迹进行细化——用人类视频提取的物体运动 T^o_t 作为任务级目标，定义物体姿态跟踪误差 e^t（平移欧式距离 + SO(3) 测地距离的加权组合），超过阈值 C 则提前终止，奖励为 r^t_obj = C − e^t。Retargeted trajectories are frequently infeasible under mismatched morphology and contact dynamics. EgoEngine refines them in simulation against an object-centric objective: the object motion T^o_t extracted from the human video serves as the task-level goal, an object-pose tracking error e^t is defined (a weighted combination of translational Euclidean distance and SO(3) geodesic distance), the episode terminates early once the error exceeds a threshold C, and the reward is r^t_obj = C − e^t.

EgoEngine 将长视域轨迹分解为时序 chunk，对每个 chunk 按能力递增顺序选择求解器：EgoEngine decomposes long-horizon trajectories into temporal chunks and, for each chunk, selects a solver in order of increasing capability:

• Replay：直接回放重定向参考轨迹，速度最快（cost = 1.00 simulation step/timestep）。Replay: directly replays the retargeted reference trajectory; the fastest option (cost = 1.00 simulation step/timestep).
• MPC（Spider）：在参考轨迹附近采样短时域动作进行局部修正，中等代价。MPC (Spider): samples short-horizon actions around the reference trajectory for local correction; moderate cost.
• RL（残差策略，PPO）：训练手部残差策略 δa_t ∼ π_φ(· | s_t)，处理接触丰富的困难 chunk，能力最强但代价最高。RL (residual policy, PPO): trains a hand residual policy δa_t ∼ π_φ(· | s_t) to handle contact-rich hard chunks; the most capable but the most expensive.

MCTS-style 策略从 Replay 开始，仅在当前模式无法满足物体中心准则时逐级升级，避免对全轨迹施以不必要的强力优化。额外采用双 chunk 联合优化窗口以避免孤立求解的局部极小值。The MCTS-style strategy starts from Replay and escalates only when the current mode fails to meet the object-centric criterion, avoiding unnecessarily heavy optimization over the entire trajectory. A joint optimization window spanning a pair of adjacent chunks is additionally adopted to avoid the local minima of solving chunks in isolation.

视觉生成（Visual Generation）Visual Generation

分三步处理每帧：(1) 人手移除（Video Inpainting）：用 SAM2 mask 遮盖手臂区域，Inpaint-Anything v2 填充被遮挡的场景与物体内容，得到无示范者帧 Ī_t；(2) 机器人渲染（Robot Rendering）：根据动作支路输出的机器人轨迹，在第一视角渲染机器人 R_t；(3) 遮挡感知融合（Occlusion-Aware Blending）：通过两次差分渲染计算可见机器人 mask M̃^r_t（保持物体不透明、机器人透明/不透明各渲染一次，对比像素差），最终合成观测：Each frame is processed in three steps: (1) Hand Removal (Video Inpainting): the arm region is masked out with the SAM2 mask and Inpaint-Anything v2 fills in the occluded scene and object content, giving the demonstrator-free frame Ī_t; (2) Robot Rendering: the robot R_t is rendered from the egocentric viewpoint according to the robot trajectory output by the action branch; (3) Occlusion-Aware Blending: the visible-robot mask M̃^r_t is computed from two differential renderings (keeping the object opaque while rendering the robot once transparent and once opaque, then comparing pixel differences), and the observation is finally composited as:

õ_t = M̃^r_t ⊙ R_t + (1 − M̃^r_t) ⊙ Ī_t

策略蒸馏（Policy Distillation）Policy Distillation

聚合所有人类视频生成的合成机器人数据集 D̃_robot = {(õ, ã)}，使用 HPT 以 ℓ₂ 动作回归损失训练 visuomotor 策略 π_θ，将双支路生成的观测与动作信息蒸馏为闭环控制器。The synthetic robot dataset D̃_robot = {(õ, ã)} generated from all human videos is aggregated, and a visuomotor policy π_θ is trained with HPT under an ℓ₂ action-regression loss, distilling the observations and actions produced by the two branches into a closed-loop controller.

03 实验Experiments

实验围绕三个问题展开：(1) 生成的机器人观测与真实机器人观测在视觉上是否一致？(2) 生成的机器人动作是否可执行且任务对齐？(3) 生成的观测–动作对是否支持零样本策略学习？使用两个数据集：TACO（2,500 段视频）和 Aria 数据集（200 段真实世界第一视角人类视频，4 个任务）。仿真机器人为双臂 RB-Y1（2×7 DoF 手臂 + 2×12 DoF XHands），真实机器人为单臂 RB-Y1。The experiments revolve around three questions: (1) are the generated robot observations visually consistent with real robot observations? (2) are the generated robot actions executable and task-aligned? (3) do the generated observation-action pairs support zero-shot policy learning? Two datasets are used: TACO (2,500 videos) and the Aria dataset (200 real-world egocentric human videos over 4 tasks). The simulated robot is a bimanual RB-Y1 (2×7 DoF arms + 2×12 DoF XHands), and the real robot is a single-arm RB-Y1.

视觉保真度（Visual Fidelity）——Fréchet Distance (FD↓) 对比Visual Fidelity - Fréchet Distance (FD↓) Comparison

EgoEngine 在 ResNet18 和 VGG16 上取得最低 FD，与真实机器人观测的特征分布最接近；DINOv2 上与 Phantom 基本持平（473.1 vs. 470.6）。定性对比中，EgoEngine 在机器人–物体接触区域与可见度排序上表现出更强的物理一致性。EgoEngine attains the lowest FD under ResNet18 and VGG16, coming closest to the feature distribution of real robot observations; under DINOv2 it is essentially on par with Phantom (473.1 vs. 470.6). Qualitatively, EgoEngine shows stronger physical consistency in robot-object contact regions and in visibility ordering.

动作保真度（Action Fidelity）——仿真成功率 SR↑ 对比Action Fidelity - Simulated Success Rate (SR↑) Comparison

EgoEngine 在 TACO 与 Aria 上均与强 RL 基线 H2S2R 持平，同时仿真开销减少约一半（TACO：34,842 vs. 73,675；Aria：16,560 vs. 20,237）。在 Aria 示范生成吞吐量上提升 22.0%（从 RL 的 2.36 demos/hour 提升至 2.88 demos/hour，单张 RTX 4090 无并行化）。EgoEngine matches the strong RL baseline H2S2R on both TACO and Aria while cutting simulation cost roughly in half (TACO: 34,842 vs. 73,675; Aria: 16,560 vs. 20,237). On Aria it raises demonstration-generation throughput by 22.0% (from 2.36 demos/hour with RL to 2.88 demos/hour, on a single RTX 4090 without parallelization).

下游策略蒸馏——真实机器人成功率（SR↑）Downstream Policy Distillation - Real-Robot Success Rate (SR↑)

EgoEngine 在 Flower 和 Hammer 两项任务上达到或超过真实机器人遥操作示范的成功率，在所有四项任务上取得非平凡的零样本性能。Human Videos 与 Phantom 基本归零，说明仅做视觉转换而不做动作细化对灵巧策略学习远不够。On the Flower and Hammer tasks EgoEngine matches or exceeds the success rate of real-robot teleoperation demonstrations, and it attains non-trivial zero-shot performance on all four tasks. Human Videos and Phantom are essentially at zero, showing that visual translation alone, without action refinement, is far from sufficient for dexterous policy learning.

消融实验（Ablation）——动作支路贡献最大Ablation - the Action Branch Contributes the Most

移除动作支路导致最大性能下降（0.51 → 0.05），印证了可执行动作生成是下游策略性能的主要因素，而视觉生成提供额外增益。仅有视觉支路时，策略在多数任务上因抓握姿态不佳而失败。Removing the action branch causes the largest performance drop (0.51 → 0.05), confirming that executable action generation is the dominant factor in downstream policy performance, while visual generation provides an additional gain. With the visual branch alone, the policy fails on most tasks because of poor grasp poses.

04 局限性Limitations