DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation

01 动机Motivation

机器人灵巧操作的数据采集极为困难：遥操作（teleoperation）因缺乏直接触觉反馈且存在空间观测不匹配，导致效率低下；而人类演示又因运动学结构、接触面形状、触觉信息和视觉外观的差异，无法直接迁移到机器人手——这一差异被称为 embodiment gap。Collecting data for robot dexterous manipulation is extremely difficult: teleoperation is inefficient because it lacks direct tactile feedback and suffers from a spatial observation mismatch; human demonstrations, in turn, cannot be transferred directly to robot hands because of differences in kinematic structure, contact surface geometry, tactile information and visual appearance—a discrepancy known as the embodiment gap.

"How can we minimize the embodiment gap, so that we can use the human hand as the universal manipulation interface for diverse robot hands?"

DexUMI framework teaser — 图1：DexUMI 框架总览。人类佩戴针对目标机器人手定制优化的外骨骼，通过与机器人手同款相机同步采集演示数据；软件流水线将外骨骼/人手视觉替换为机器人手图像，消除视觉域差距，最终用于策略训练。Figure 1: Overview of the DexUMI framework. A human wears an exoskeleton customized and optimized for the target robot hand, and collects demonstration data synchronously with the same camera model used on the robot hand; the software pipeline replaces the exoskeleton/human hand in the visual stream with robot hand images, eliminating the visual domain gap, and the result is finally used for policy training.

86%两平台平均任务成功率Average task success rate across both platforms

3.2×相对遥操作的数据采集效率提升Data collection efficiency gain over teleoperation

2支持的机器人手平台（Inspire + XHand）Supported robot hand platforms (Inspire + XHand)

5独立评测任务数量Independent evaluation tasks

02 方法Method

DexUMI 通过两条并行路径弥合 embodiment gap：硬件适应——设计针对目标机器人手双层优化的可穿戴外骨骼，使运动学工作空间高度匹配；软件适应——构建视觉流水线，将演示视频中的外骨骼/人手无缝替换为机器人手图像，消除视觉分布差异。DexUMI bridges the embodiment gap along two parallel paths: hardware adaptation—a wearable exoskeleton bi-level optimized for the target robot hand, so that the kinematic workspaces match closely; software adaptation—a visual pipeline that seamlessly replaces the exoskeleton/human hand in the demonstration videos with robot hand images, eliminating the visual distribution shift.

硬件适应：外骨骼设计与双层优化Hardware adaptation: exoskeleton design and bi-level optimization

图2：外骨骼硬件设计。关节编码器（joint encoders）捕获手指弯曲角度，腕部通过 iPhone ARKit 追踪 6-DoF 位姿，同步相机（150° DFoV）与机器人手相机位置完全对应，FSR 传感器提供触觉反馈。Figure 2: Exoskeleton hardware design. Joint encoders capture finger flexion angles, the wrist 6-DoF pose is tracked with iPhone ARKit, the synchronized camera (150° DFoV) sits exactly where the robot hand camera does, and FSR sensors provide tactile feedback.

外骨骼并非通用设计，而是针对每款目标机器人手通过 bi-level workspace matching 优化参数。优化目标为最大化外骨骼与机器人手工作空间的双向相似度：第一项鼓励外骨骼覆盖机器人手的可达工作空间；第二项约束外骨骼生成的动作落在机器人手能力范围之内（⊆ 约束），防止产生不可复现的姿态。拇指采用专项机制设计以避免碰撞同时保留指尖映射精度。The exoskeleton is not a universal design; its parameters are optimized for each target robot hand through bi-level workspace matching. The objective maximizes the bidirectional similarity between the workspaces of the exoskeleton and the robot hand: the first term encourages the exoskeleton to cover the reachable workspace of the robot hand; the second term constrains the actions generated by the exoskeleton to fall within the capability of the robot hand (a ⊆ constraint), preventing unreproducible poses. The thumb uses a dedicated mechanism design that avoids collisions while preserving fingertip mapping accuracy.

软件适应：Robot Hand Inpainting 视觉流水线Software adaptation: the Robot Hand Inpainting visual pipeline

Software visual adaptation pipeline — 图4：软件视觉适应流水线。依次经过 SAM2 分割（segmentation）提取外骨骼/人手掩码，ProPainter 基于光流的 inpainting 恢复背景，再将预先录制的机器人手图像与原始背景通过 occlusion-aware compositing 合成，最终生成视觉上与真实机器人演示一致的训练视频。Figure 4: The software visual adaptation pipeline. SAM2 segmentation first extracts the exoskeleton/human hand mask, ProPainter restores the background with optical-flow-based inpainting, and pre-recorded robot hand images are then composited with the original background through occlusion-aware compositing, finally producing training videos visually consistent with real robot demonstrations.

视觉流水线分四步：① 使用 SAM2 对每帧进行手部分割；② 使用 ProPainter 进行光流引导的背景修复（inpainting）；③ 使用真实机器人手按照采集动作序列录制对应视角视频；④ 采用 occlusion-aware compositing 将机器人手自然融合到恢复背景中，保留物体被遮挡的自然关系。策略动作采用 relative trajectory（相对轨迹）而非绝对坐标，以提升对硬件噪声的鲁棒性。The visual pipeline has four steps: ① SAM2 segments the hand in every frame; ② ProPainter performs optical-flow-guided background inpainting; ③ the real robot hand replays the collected action sequence to record video from the corresponding viewpoint; ④ occlusion-aware compositing blends the robot hand naturally into the restored background, preserving the natural occlusion relationship with the objects. Policy actions use a relative trajectory rather than absolute coordinates, improving robustness to hardware noise.

触觉集成Tactile integration

外骨骼集成 FSR（力敏电阻）传感器，其布局与目标机器人手的触觉传感器对应，直接采集接触力信号，用于需精细力控的任务策略学习。实验表明触觉反馈对采用 relative action 的策略有显著帮助。The exoskeleton integrates FSR (force-sensitive resistor) sensors whose layout corresponds to the tactile sensors of the target robot hand, directly collecting contact force signals for policy learning on tasks that require fine force control. Experiments show that tactile feedback helps significantly for policies using relative action.

数据采集效率Data collection efficiency

在 15 分钟的采集会话中，DexUMI 达到传统遥操作方法 3.2 倍的采集效率（以 tea picking using tool 任务为基准）。采集的轨迹数量：Cube Picking 310 条，Egg Carton 175 条，Tea Picking 400 条，Kitchen 370+100 条。In a 15-minute collection session, DexUMI reaches 3.2 times the collection efficiency of conventional teleoperation (benchmarked on the tea picking using tool task). Numbers of trajectories collected: Cube Picking 310, Egg Carton 175, Tea Picking 400, Kitchen 370+100.

03 实验Experiments

实验在两款商用灵巧手（Inspire Hand 和 XHand）上评测，涵盖 5 个任务：Cube Picking、Egg Carton Opening、Tea Picking（工具/散叶）、厨房综合任务（Knob/Pan/Salt）。策略基于 Diffusion Policy，评估指标为任务成功率（success rate）。Experiments are evaluated on two commercial dexterous hands (Inspire Hand and XHand) and cover 5 tasks: Cube Picking, Egg Carton Opening, Tea Picking (tool/loose leaf) and a composite kitchen task (Knob/Pan/Salt). Policies are based on Diffusion Policy, and the evaluation metric is the task success rate.

任务成功率汇总Task success rate summary

任务Task	平台Platform	成功率Success rate
Cube Picking	Inspire Hand	1.00
Egg Carton Opening	Inspire Hand	0.85
Tea Picking (tool)	Inspire Hand	1.00
Tea Picking (leaf)	Inspire Hand	0.85
Tea Picking (tool)	XHand	1.00
Tea Picking (leaf)	XHand	0.85
Kitchen — Knob	XHand	0.95
Kitchen — Pan	XHand	0.75
Kitchen — Salt	XHand	0.75

Policy evaluation results across tasks — 图5：各任务策略评测结果。覆盖 Cube Picking、Egg Carton Opening、Tea Picking（工具与散叶）和厨房综合任务（Knob、Pan、Salt），在 Inspire Hand 和 XHand 两平台上均验证了 DexUMI 的有效性。Figure 5: Policy evaluation results on each task. Covering Cube Picking, Egg Carton Opening, Tea Picking (tool and loose leaf) and the composite kitchen task (Knob, Pan, Salt), the effectiveness of DexUMI is validated on both the Inspire Hand and XHand platforms.

视觉适应消融：Inpainting 的关键作用Visual adaptation ablation: the critical role of inpainting

视觉输入方式Visual input	Cube Picking	Egg Carton	Tea (tool)
Raw image（原始图像）Raw image	0.20	0.05	0.85
Masked image（掩码图像）Masked image	0.60	0.10	0.90
DexUMI Inpainting	1.00	0.85	1.00

Ablation study: relative vs absolute actions and tactile — 图6：消融实验。Relative action（相对轨迹）在所有任务上均优于 absolute action；触觉反馈仅对 relative action 策略有提升效果，对 absolute action 策略反而降低性能——表明触觉信号的有效利用依赖于鲁棒的动作表示。Figure 6: Ablation study. Relative action outperforms absolute action on all tasks; tactile feedback improves only the relative action policies and instead degrades the absolute action policies—showing that effective use of tactile signals depends on a robust action representation.

关键消融结论Key ablation findings

Relative vs. Absolute Action：相对轨迹表示在所有任务上均优于绝对坐标，对硬件噪声具有更强鲁棒性。Relative vs. Absolute Action: the relative trajectory representation outperforms absolute coordinates on all tasks and is more robust to hardware noise.
触觉感知：仅 relative action 策略从触觉反馈中受益；absolute action 策略加入触觉后性能下降，说明有噪声的触觉信号对动作鲁棒性有要求。Tactile sensing: only relative action policies benefit from tactile feedback; absolute action policies degrade once tactile input is added, indicating that noisy tactile signals demand a robust action representation.
视觉 Inpainting：Inpainting 相比 masked 和 raw 基线有显著提升（Cube：1.00 vs. 0.60 vs. 0.20），是视觉域适应的关键组件。Visual inpainting: inpainting improves markedly over the masked and raw baselines (Cube: 1.00 vs. 0.60 vs. 0.20) and is the key component of visual domain adaptation.

04 局限性Limitations

说明：以下局限性均为作者在论文中明确陈述（stated）的已知问题。Note: all limitations below are known issues explicitly stated by the authors in the paper.

外骨骼需针对每款机器人手定制The exoskeleton must be customized for each robot hand

尽管提出了双层优化框架，外骨骼的机械设计仍需针对每款目标机器人手进行适配，无法做到完全通用。3D 打印材料在人手力的作用下可能发生形变，影响关节编码器的测量精度。Although a bi-level optimization framework is proposed, the mechanical design of the exoskeleton still has to be adapted to each target robot hand and cannot be made fully universal. 3D printed material may deform under the forces of the human hand, affecting the measurement accuracy of the joint encoders.

工作空间覆盖局限于指尖，忽略掌心接触Workspace coverage is limited to the fingertips and ignores palm contact

当前的工作空间匹配优化专注于指尖轨迹，未考虑掌心（palm）接触几何，限制了需要整手接触的操作任务的适用范围。The current workspace matching optimization focuses on fingertip trajectories and does not consider palm contact geometry, which limits its applicability to manipulation tasks requiring whole-hand contact.

传感器可靠性问题Sensor reliability issues

FSR 触觉传感器对贴附方式敏感，高压下电磁传感器存在漂移（drift）问题。机器人关节的间隙（backlash）和摩擦导致编码器精度仅在单方向上可靠。FSR tactile sensors are sensitive to how they are attached, and the electromagnetic sensor drifts under high pressure. Backlash and friction in the robot joints make the encoder accurate only in a single direction.

软件流水线依赖真实硬件且存在光照不一致问题The software pipeline depends on real hardware and suffers from lighting inconsistency

生成机器人手图像需要真实机器人硬件配合录制，无法完全离线生成。Inpainting 在光照条件差异较大时存在伪影（artifacts）。相机固定于手部，不支持移动相机视角。Generating robot hand images requires recording with real robot hardware and cannot be done fully offline. Inpainting produces artifacts when lighting conditions differ substantially. The camera is fixed to the hand, so moving camera viewpoints are not supported.