UMI — Universal Manipulation Interface

01 动机Motivation

机器人操作策略的训练数据严重不足：远程操控需要昂贵硬件和专家操作员，而直接利用人类在野外的视频又存在巨大的动作域差（embodiment gap）。现有手持夹爪方案往往只能处理简单的拾取任务，无法支持动态投掷、双臂折叠或长时序洗碗等复杂操作。Training data for robot manipulation policies is severely scarce: teleoperation demands expensive hardware and expert operators, while directly exploiting in-the-wild human video suffers from a large embodiment gap. Existing hand-held gripper designs can usually handle only simple picking tasks, and cannot support complex manipulations such as dynamic tossing, bimanual cloth folding or long-horizon dish washing.

"We identify four core issues that prevent direct action transfer from human demonstration to robot execution: insufficient visual context, action imprecision, latency discrepancies, and insufficient policy representations for multimodal action distributions."

UMI teaser — 图1：UMI 框架总览。研究者手持配备鱼眼相机和侧面反射镜的夹爪，在咖啡馆、户外等真实场景采集演示数据，最终训练出可在 UR5、Franka FR2 等多款机器人上直接部署的策略，涵盖杯子摆放（Cup Arrangement）、动态投掷（Dynamic Tossing）、双臂折衣（Bimanual Cloth Folding）和洗碗（Dish Washing）四项任务。Figure 1: Overview of the UMI framework. Researchers hand-hold a gripper equipped with a fisheye camera and side mirrors to collect demonstration data in real scenes such as cafes and outdoors, ultimately training policies that deploy directly on multiple robots including UR5 and Franka FR2, covering four tasks: Cup Arrangement, Dynamic Tossing, Bimanual Cloth Folding and Dish Washing.

20/20杯子摆放满分成功率（Full UMI）perfect success rate on Cup Arrangement (Full UMI)

87.5%动态投掷成功率（105/120）Dynamic Tossing success rate (105/120)

70%7步洗碗任务成功率（14/20）success rate on the 7-step Dish Washing task (14/20)

$371单套夹爪总硬件成本total hardware cost of one gripper

02 方法Method

UMI 分为两个设计层次：演示接口（Demonstration Interface, HD1–HD6）解决数据采集中的观测与动作精度问题；策略接口（Policy Interface, PD1–PD2）解决训练到推理的延迟和坐标系不一致问题。两者共同支撑以 Diffusion Policy 为骨干的硬件无关策略。UMI is organized into two design layers: the Demonstration Interface (HD1–HD6) resolves observation and action-precision issues during data collection; the Policy Interface (PD1–PD2) resolves the latency and coordinate-frame mismatches between training and inference. Together they support a hardware-agnostic policy built on Diffusion Policy as its backbone.

UMI hardware design — 图2：手持夹爪硬件设计。左：155° 鱼眼 GoPro + 侧面反射镜 + 柔性手指 + fiducial markers（用于连续夹爪宽度跟踪）。右：夹爪可直接安装到 UR5、Franka 等末端执行器。总重 780g，3D 打印成本 $73，GoPro 套装 $298。Figure 2: Hand-held gripper hardware design. Left: 155° fisheye GoPro + side mirrors + soft fingers + fiducial markers (for continuous gripper-width tracking). Right: the gripper mounts directly onto end-effectors such as UR5 and Franka. Total weight 780g, 3D printing cost $73, GoPro kit $298.

演示接口设计（HD1–HD6）Demonstration Interface Design (HD1–HD6)

HD1–HD2 · 腕部鱼眼相机HD1–HD2 · Wrist-mounted Fisheye Camera

单相机安装于夹爪腕部，消除外部相机的标定需求，并通过相机运动自然实现数据多样化。采用 155° 鱼眼镜头（raw fisheye，不做矫正），在保留中心分辨率的同时压缩边缘信息，策略学习效果优于等效pinhole模型（消融：去掉鱼眼后杯子任务从100%降至55%）。A single camera mounted at the gripper wrist removes the calibration requirement of external cameras, and camera motion naturally diversifies the data. A 155° fisheye lens is used (raw fisheye, without rectification), compressing peripheral information while preserving central resolution; policy learning outperforms an equivalent pinhole model (ablation: removing the fisheye drops the cup task from 100% to 55%).

HD3 · 侧面反射镜HD3 · Side Mirrors

在夹爪两侧放置物理反射镜，使单张图像中隐含立体视角（implicit stereo）。训练时对镜像内容做数字翻转（digital reflection），获得最优策略效果（消融：去掉镜面翻转后成功率从100%降至85%）。Physical mirrors placed on both sides of the gripper embed an implicit stereo view within a single image. During training the mirrored content undergoes digital reflection, which yields the best policy performance (ablation: removing the mirror flip drops the success rate from 100% to 85%).

HD4 · IMU 感知 SLAMHD4 · IMU-aware SLAM

将 GoPro 内置 IMU 与 ORB-SLAM3 结合，实现视觉-惯性 SLAM。即使存在运动模糊也能保持追踪，并恢复具有度量尺度（metric scale）的动作——这对动态投掷等任务至关重要。SLAM 位置精度：6.1mm 平均轨迹误差，旋转精度：3.5°。The GoPro's built-in IMU is combined with ORB-SLAM3 to realize visual-inertial SLAM. Tracking is preserved even under motion blur, and actions are recovered with metric scale — essential for tasks such as dynamic tossing. SLAM position accuracy: 6.1mm mean trajectory error; rotation accuracy: 3.5°.

HD5–HD6 · 连续夹爪控制 + 运动学过滤HD5–HD6 · Continuous Gripper Control + Kinematic Filtering

通过 fiducial markers 连续跟踪夹爪宽度（而非二值开/闭），实现精确时序控制和隐式力控（soft-finger deformation）。训练时通过运动学可行性过滤，将机器人无关的演示数据适配到具体机器人。Gripper width is tracked continuously through fiducial markers (rather than binary open/close), enabling precise temporal control and implicit force control via soft-finger deformation. During training, kinematic feasibility filtering adapts robot-agnostic demonstration data to a specific robot.

策略接口设计（PD1–PD2）Policy Interface Design (PD1–PD2)

Policy interface: latency matching and relative trajectory — 图5：策略接口设计。左：推理时延迟匹配——分别测量 RGB、本体感知、夹爪各路信号延迟，推理时将所有流对齐至延迟最高的相机；动作延迟通过提前发送指令补偿。右：相对轨迹动作表示——动作为相对初始末端执行器位姿的 SE(3) 变换，本体感知表示为相对轨迹（等效速度），双臂相对位姿通过 map-then-localize SLAM 计算。Figure 5: Policy interface design. Left: inference-time latency matching — the latency of the RGB, proprioception and gripper channels is measured separately, and at inference all streams are aligned to the camera with the highest latency; action latency is compensated by issuing commands ahead of time. Right: the relative trajectory action representation — actions are SE(3) transforms relative to the initial end-effector pose, proprioception is expressed as a relative trajectory (equivalent to velocity), and the bimanual relative pose is computed by map-then-localize SLAM.

PD1.1 观测延迟匹配：逐路测量 RGB、本体感知、夹爪延迟，推理时通过时间插值对齐到最高延迟的相机流。PD1.1 Observation latency matching: the latency of the RGB, proprioception and gripper channels is measured channel by channel, and at inference the streams are aligned by temporal interpolation to the highest-latency camera stream.
PD1.2 动作延迟补偿：将指令提前发送，抵消传感器→推理→执行的完整延迟链路（消融：去掉延迟匹配后投掷任务成功率从87.5%跌至57.5%）。PD1.2 Action latency compensation: commands are issued in advance to cancel the full sensor → inference → execution latency chain (ablation: removing latency matching drops the tossing success rate from 87.5% to 57.5%).
PD2.1–2.2 相对末端执行器轨迹：动作表示为相对初始位姿的 SE(3) 变换，本体感知表示为相对轨迹（2帧 horizon，即速度），系统无需机器人基座标定（消融：绝对动作表示仅25%成功率，delta 动作80%，相对轨迹100%）。PD2.1–2.2 Relative end-effector trajectory: actions are represented as SE(3) transforms relative to the initial pose and proprioception as a relative trajectory (2-frame horizon, i.e. velocity), so the system needs no robot base calibration (ablation: the absolute action representation reaches only 25% success, delta actions 80%, relative trajectory 100%).
PD2.3 双臂相对位姿：通过 map-then-localize 建立共享场景坐标系，计算夹爪间相对位姿（消融：去掉此项后双臂折衣成功率从70%降至30%）。PD2.3 Bimanual relative pose: map-then-localize establishes a shared scene coordinate frame in which the relative pose between grippers is computed (ablation: removing it drops bimanual cloth folding from 70% to 30%).

策略骨干网络采用 Diffusion Policy，专为建模多峰动作分布设计，适应野外数据中自然存在的多模态行为。The policy backbone is Diffusion Policy, designed specifically to model multimodal action distributions and thus suited to the multimodal behaviour naturally present in in-the-wild data.

03 实验Experiments

实验分为两类：窄域能力验证（单场景，含完整消融）和野外泛化（30个多样场景，15种杯子）。机器人平台：UR5 + Franka FR2；演示采集：每任务250–305条；评估：每条件20次（投掷120次）。Experiments fall into two groups: narrow-domain capability validation (single scene, with full ablations) and in-the-wild generalization (30 diverse scenes, 15 kinds of cups). Robot platforms: UR5 + Franka FR2; demonstration collection: 250–305 episodes per task; evaluation: 20 trials per condition (120 for tossing).

窄域（Narrow-Domain）能力评估Narrow-Domain Capability Evaluation

任务Task	演示数量Demonstrations	Full UMI	关键消融对比Key ablation comparison
Cup Arrangement（杯子摆放）Cup Arrangement	305	20/20 (100%)	无鱼眼: 55% · delta动作: 80% · 绝对动作: 25%no fisheye: 55% · delta action: 80% · absolute action: 25%
Dynamic Tossing（动态投掷）Dynamic Tossing	280	105/120 (87.5%)	无延迟匹配: 57.5%no latency matching: 57.5%
Bimanual Cloth Folding（双臂折衣）Bimanual Cloth Folding	250	14/20 (70%)	无双臂相对位姿: 30%no bimanual relative pose: 30%
Dish Washing（7步洗碗）Dish Washing (7 steps)	258	14/20 (70%)	ResNet-34（无CLIP）: 0%ResNet-34 (no CLIP): 0%

图8：窄域评估结果。每列显示初始状态叠加图、典型失败案例和成功率条形图，同时对比各消融条件（无鱼眼、无延迟匹配、无双臂相对位姿等）。数字均来自论文原文，Full UMI 在四项任务上均显著优于所有消融变体。Figure 8: Narrow-domain evaluation results. Each column shows the overlaid initial states, a representative failure case and a success-rate bar chart, alongside the corresponding ablation conditions (no fisheye, no latency matching, no bimanual relative pose, and so on). All numbers are taken from the paper; Full UMI clearly outperforms every ablated variant on all four tasks.

野外泛化（In-the-Wild Generalization）In-the-Wild Generalization

以 Cup Arrangement 为泛化测试任务，在 30 个多样场景（家居、办公室、餐厅、户外）共采集 1,400 条演示，涵盖 15 种不同杯子，采集时间约 12 人时。Cup Arrangement serves as the generalization test task: 1,400 demonstrations were collected across 30 diverse scenes (homes, offices, restaurants, outdoors), covering 15 different cups, in roughly 12 person-hours.

测试环境Test environment	已见杯子Seen cups	未见杯子Unseen cups	合并Combined
咖啡桌（Cafe table，未见场景）Cafe table (unseen scene)	28/40 (70%)	15/20 (75%)	43/60 (71.7%)
饮水机旁（Water fountain，分布外）Water fountain (out-of-distribution)	包含在合并结果中included in the combined result		71.7%
窄域对照基线（同预训练权重）Narrow-domain baseline (same pretrained weights)	0%（在新环境完全失败）0% (complete failure in the new environment)

图9：野外泛化结果。左侧展示在咖啡馆、户外等多样场景中的成功执行；右侧对比在未见场景中 UMI 野外策略（~70–75%）与窄域基线（0%）的成功率。Figure 9: In-the-wild generalization results. Left: successful executions in diverse scenes such as cafes and outdoors. Right: success rates in unseen scenes for the UMI in-the-wild policy (~70–75%) versus the narrow-domain baseline (0%).

数据采集效率与 SLAM 精度Data Collection Efficiency and SLAM Accuracy

UMI 采集速度达人手速度的 48%，为 SpaceMouse 遥控的 3×；动态投掷达人手速度 64%，SpaceMouse 完全无法操控。UMI collects at 48% of bare-hand speed, 3× that of SpaceMouse teleoperation; for dynamic tossing it reaches 64% of bare-hand speed, a task SpaceMouse cannot perform at all.
SLAM 位置误差：6.1mm MAE；旋转误差：3.5°；双臂相对位姿：位置 10.1mm，旋转 0.8°（MoCap 基准，14 任务平均）。SLAM position error: 6.1mm MAE; rotation error: 3.5°; bimanual relative pose: 10.1mm in position, 0.8° in rotation (MoCap reference, averaged over 14 tasks).

跨机器人迁移Cross-Robot Transfer

UR5 上采集的杯子摆放演示直接迁移至 Franka FR2，成功率 18/20 (90%)，验证了硬件无关策略的零样本跨平台迁移能力。Cup Arrangement demonstrations collected on UR5 transfer directly to Franka FR2 with a success rate of 18/20 (90%), validating the zero-shot cross-platform transfer of the hardware-agnostic policy.

04 局限性Limitations

注：以下局限性均来自论文作者明确陈述（stated），最后一条为从系统设计推断（inferred）。Note: all limitations below are explicitly stated by the authors of the paper (stated); the last one is inferred from the system design (inferred).

SLAM 依赖环境纹理SLAM depends on environmental texture

ORB-SLAM3 在纹理匮乏的场景（如纯白墙面、低光照环境）下追踪失败，导致无法恢复度量尺度动作。作者明确指出这是当前方案的主要瓶颈之一。ORB-SLAM3 loses tracking in texture-poor scenes (plain white walls, low-light environments), so metric-scale actions cannot be recovered. The authors explicitly identify this as one of the main bottlenecks of the current design.

训练时需要运动学可行性过滤Kinematic feasibility filtering is required at training time

采集的人类演示数据需要在训练阶段针对具体机器人执行运动学可行性过滤（kinematic feasibility filtering），系统尚未实现端到端的体态感知学习（embodiment-aware learning）。The collected human demonstration data must undergo kinematic feasibility filtering for the specific robot during training; the system does not yet achieve end-to-end embodiment-aware learning.

手持夹爪的自由度和速度受限The hand-held gripper is limited in degrees of freedom and speed

780g 的夹爪重量和两指结构限制了采集效率（仅达人手速度 48%）及可表达的操作多样性，与人手相比在灵巧度上有明显差距。The 780g gripper weight and the two-finger structure limit collection efficiency (only 48% of bare-hand speed) and the diversity of manipulations that can be expressed, leaving a clear dexterity gap relative to the human hand.

[推断] 策略泛化依赖演示数据多样性[Inferred] Policy generalization depends on demonstration data diversity

野外泛化实验表明，策略的跨场景迁移能力直接依赖于演示数据覆盖的场景和物体多样性；当测试场景与演示分布差异过大时，成功率会显著下降（inferred from 实验设计）。The in-the-wild generalization experiments show that cross-scene transfer ability depends directly on the scene and object diversity covered by the demonstration data; when the test scene deviates too far from the demonstration distribution, the success rate drops markedly (inferred from the experimental design).