RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation

01 动机 MotivationMotivation

通用机器人操作策略的训练需要大量多样化、高质量的示范数据。然而现有数据集普遍存在规模有限、机体单一、采集标准不统一等问题，严重制约了策略的泛化能力。Training general-purpose robot manipulation policies requires large amounts of diverse, high-quality demonstration data. Existing datasets, however, commonly suffer from limited scale, a single embodiment and inconsistent collection standards, which severely constrains the generalization ability of the learned policies.

"Unlike language or vision datasets that can often be sourced through web-based collection methods, collecting robotic data is difficult because each robot requires controlled environments where the joints and end-effector information of robotic systems are meticulously recorded."

此前大多数工作仅专注于单一机器人类型（如 Open X-Embodiment 以单臂为主），任务多样性和跨机体迁移能力严重受限。RoboMIND 提出了一套统一数据采集与标注流程，汇聚 4 类机体的真实遥操作数据，并额外收录失败案例和仿真数字孪生，以满足当代大模型训练的数据需求。Most prior work focuses on a single robot type (e.g. Open X-Embodiment is dominated by single-arm platforms), which severely limits task diversity and cross-embodiment transfer. RoboMIND proposes a unified data collection and annotation pipeline that aggregates real teleoperation data from 4 embodiments, and additionally includes failure cases and a simulated digital twin, so as to meet the data demands of training contemporary large models.

与现有工作相比，RoboMIND 的优势体现在：(1) 覆盖的任务和机体类型最多；(2) 采集标准统一，数据质量可控；(3) 提供失败轨迹以支持 Reinforcement Learning from Human Feedback (RLHF) 类研究；(4) 提供 Isaac Sim 数字孪生环境，支持合成数据生成与策略评测。论文 Table I 与 Open X-Embodiment、BridgeData V2、RoboSet、DROID 等代表性数据集进行了详细对比。Compared with existing work, RoboMIND offers: (1) the broadest coverage of tasks and embodiment types; (2) a unified collection standard with controllable data quality; (3) failure trajectories that support Reinforcement Learning from Human Feedback (RLHF) style research; (4) an Isaac Sim digital-twin environment supporting synthetic data generation and policy evaluation. Table I of the paper gives a detailed comparison with representative datasets such as Open X-Embodiment, BridgeData V2, RoboSet and DROID.

02 方法 MethodMethod

RoboMIND 的核心贡献是一套端到端的数据采集、处理与标注流程，以及配套的数字孪生评测环境，确保跨机体数据的一致性与可用性。The core contribution of RoboMIND is an end-to-end pipeline for data collection, processing and annotation, together with a matching digital-twin evaluation environment, ensuring the consistency and usability of data across embodiments.

数据采集与存储（Data Collection & Storage）Data Collection & Storage

所有数据均通过人工遥操作采集，遵循统一的 H5 格式存储：每条轨迹包含多视角 RGB-D 图像、本体感知机器人状态（关节角度、末端执行器位姿）和自然语言任务描述。四类机体使用各自定制化的遥操作装置：Franka 使用 3D 打印主臂（39.2% 轨迹总量），Tien Kung 使用动捕套装（15,187 条），AgileX 使用双侧主臂（25,170 条），URS 使用仿真（500 条）。All data are collected by human teleoperation and stored in a unified H5 format: each trajectory contains multi-view RGB-D images, proprioceptive robot states (joint angles, end-effector poses) and a natural-language task description. The four embodiments use their own customized teleoperation devices: Franka uses a 3D-printed leader arm (39.2% of all trajectories), Tien Kung uses a motion-capture suit (15,187 trajectories), AgileX uses bilateral leader arms (25,170 trajectories) and URS uses simulation (500 trajectories).

数据预处理与质检（Data Preprocessing & QA）Data Preprocessing & QA

采集后的数据经过三阶段质检：(1) 快速检视——确认无明显技术问题（帧丢失、冻结）；(2) 详细检视——逐帧回放确认操作质量；(3) 类别标注——在时间戳上标注不合规数据并分类原因。论文定义了 8 类质量缺陷（如 Touch Excess、Movement not Smooth、Collision before Grabbing、Image Distortion 等），所有不合格片段均标注上下文和原因供后续分析。The collected data pass a three-stage quality check: (1) quick inspection -- confirming there is no obvious technical problem (dropped or frozen frames); (2) detailed inspection -- frame-by-frame playback to confirm manipulation quality; (3) category annotation -- marking non-compliant data on the timeline and classifying the cause. The paper defines 8 categories of quality defect (such as Touch Excess, Movement not Smooth, Collision before Grabbing and Image Distortion), and every non-compliant segment is annotated with its context and cause for later analysis.

任务分类与语言标注（Task Classification & Language Annotation）Task Classification & Language Annotation

所有任务按语义分为 6 大类：Articulated Manipulations（Artic. M.）（如开关抽屉）、Coordination Manipulations（Coord. M.）（双臂协调）、Basic Manipulations（Basic M.）（抓放）、Multiple Object Interactions（Obj. Int.）、Precision Manipulations（M. Precision）、Scene Understandings（Scene U.）。每条轨迹都配备由 Gemini 初步生成、人工精修的分步式语言描述，精确描述每个动作段落。All tasks are grouped into 6 semantic categories: Articulated Manipulations (Artic. M.) (e.g. opening and closing drawers), Coordination Manipulations (Coord. M.) (dual-arm coordination), Basic Manipulations (Basic M.) (pick and place), Multiple Object Interactions (Obj. Int.), Precision Manipulations (M. Precision) and Scene Understandings (Scene U.). Every trajectory comes with a step-wise language description first generated by Gemini and then refined by humans, describing each action segment precisely.

数字孪生环境（Isaac Sim Digital Twin）Isaac Sim Digital Twin

论文同步构建了 Isaac Sim 仿真数字孪生，复现真实机器人平台的外观、动力学及相机配置，支持合成数据生成（500 条 URS 仿真轨迹）和策略在仿真中的系统性评测，降低真实环境部署成本。The paper also builds an Isaac Sim digital twin that reproduces the appearance, dynamics and camera configuration of the real robot platforms, supporting synthetic data generation (500 simulated URS trajectories) and systematic evaluation of policies in simulation, lowering the cost of real-world deployment.

03 实验 ExperimentsExperiments

论文通过两大实验系列验证 RoboMIND 的价值：(1) 在单任务模仿学习算法上的 benchmark 评测；(2) 将 RoboMIND 用于微调大型 VLA 模型的泛化性验证。共选取 45 项任务（含单臂、双臂、人形）进行真实机器人测试，每项任务运行 10 次取成功率。The paper validates the value of RoboMIND through two series of experiments: (1) a benchmark evaluation of single-task imitation learning algorithms; (2) a generalization study that uses RoboMIND to fine-tune large VLA models. In total 45 tasks (single-arm, dual-arm and humanoid) are tested on real robots, each task being run 10 times to obtain a success rate.

基准算法对比（Single-Task Imitation Learning）Single-Task Imitation Learning

在 RoboMIND 上评测了三种算法：Three algorithms are evaluated on RoboMIND:

• ACT（Action Chunking with Transformers，Zhao et al.）ACT (Action Chunking with Transformers, Zhao et al.)
• Diffusion Policy（扩散策略，Chi et al.）Diffusion Policy (diffusion-based policy, Chi et al.)
• BAKU（多任务行为学习，Haldar et al.）BAKU (multi-task behavior learning, Haldar et al.)

每类算法在各机体上的 15 个任务进行测试，评估指标为任务成功率（success rate）。结果显示 ACT 在大多数任务上表现最优，平均成功率达 55.3%（跨 45 任务）。Each algorithm is tested on 15 tasks per embodiment, with task success rate as the evaluation metric. ACT performs best on most tasks, reaching an average success rate of 55.3% (across 45 tasks).

注：上表数值为论文 Fig. 12 可视化图中读取的近似值。精确的逐任务成功率请参见原文 Table IV–VI。Note: The values in the table above are approximations read off the visualization in Fig. 12 of the paper. See Tables IV-VI of the original paper for the exact per-task success rates.

VLA 模型微调与泛化性验证VLA Fine-tuning and Generalization Study

论文选取三种 VLA 大模型：OpenVLA、RDT-1B、CrossFormer，分别在 RoboMIND 全量数据上微调，并在 Franka 单臂的 45 个任务上测试成功率。The paper selects three large VLA models -- OpenVLA, RDT-1B and CrossFormer -- fine-tunes each of them on the full RoboMIND data, and measures success rates on 45 Franka single-arm tasks.

实验结论（论文 Table IV–VI）：Experimental findings (Tables IV-VI of the paper):

• 使用完整 RoboMIND 微调后，RDT-1B 和 CrossFormer 在单臂操作任务上相比原始预训练权重取得显著提升。After fine-tuning on the complete RoboMIND, RDT-1B and CrossFormer improve significantly over their original pre-trained weights on single-arm manipulation tasks.
• OpenVLA 直接在专项多任务数据集上微调，在 FR-PlaceBreadPlate 等复杂任务上成功率超过 Franka（70% vs. 50%）。OpenVLA, fine-tuned directly on the dedicated multi-task dataset, exceeds Franka on complex tasks such as FR-PlaceBreadPlate (70% vs. 50%).
• 人形数据（Tien Kung）的加入有助于提升模型对双臂协调任务（Bimanual Tasks）的理解，跨机体迁移显示出初步可行性。Adding humanoid data (Tien Kung) helps the model understand dual-arm coordination tasks (Bimanual Tasks) better, and cross-embodiment transfer shows preliminary feasibility.
• 仿真-真实联合训练实验（Co-training with Real and Simulation Data）显示：在 EM-UprightCup 等复杂任务上，加入 100 条真实轨迹 + 500 条仿真轨迹可接近单纯使用 200 条真实轨迹的效果，验证了数字孪生的辅助价值。The co-training experiments (Co-training with Real and Simulation Data) show that on complex tasks such as EM-UprightCup, adding 100 real trajectories plus 500 simulated trajectories approaches the performance of using 200 real trajectories alone, confirming the auxiliary value of the digital twin.
• 泛化测试（unseen objects & backgrounds）中 ACT 在未见物体上平均成功率约 40.0%（F2），未见背景上约 48.0%（F2），表明仍有较大提升空间。In the generalization tests (unseen objects & backgrounds), ACT reaches an average success rate of about 40.0% (F2) on unseen objects and about 48.0% (F2) on unseen backgrounds, showing that there is still considerable room for improvement.

失败原因分析（Failure Case Analysis）Failure Case Analysis

对 ACT 在 45 项任务的失败进行分类（论文 Fig. 15），最常见的前 5 类失败原因包括：Inaccurate Positioning（~48%，人形机体最高）、Early Release、Cannot Close Gripper、Object Drop、Cannot Return to Home Pose。失败案例分析数据有助于指导后续数据采集和策略改进方向。Classifying ACT's failures across the 45 tasks (Fig. 15 of the paper), the 5 most common causes are: Inaccurate Positioning (~48%, highest on the humanoid embodiment), Early Release, Cannot Close Gripper, Object Drop and Cannot Return to Home Pose. This failure analysis helps guide future data collection and directions for improving policies.

04 局限性 LimitationsLimitations