数据驱动的 imitation learning 已深刻改变机器人操作领域，但现有方法受制于大规模、多样化真实演示数据的匮乏，
在长时域双臂任务与陌生环境移动操作中泛化能力依然不足。Data-driven imitation learning has profoundly reshaped robotic manipulation, yet existing approaches remain constrained by the scarcity of large-scale, diverse real-world demonstration data, and still generalize poorly on long-horizon bimanual tasks and on mobile manipulation in unfamiliar environments.
"Data-driven imitation learning has revolutionized robotic manipulation, but existing approaches remain limited by
the scarcity of large-scale, diverse real-world demonstration data, leading to insufficient generalization in
long-horizon bimanual tasks and mobile manipulation in strange environments."
现有代表性数据集存在明显的单一维度局限：BridgeData V2（60K 轨迹，仅 13 项技能）、DROID（76K 轨迹，仅含单臂 Franka）、
Open X-Embodiment（数据聚合，无触觉）。即便是最新的 AgiBot World（1M 轨迹）和 Galaxea Open-World（50K），
也均依赖单一机器人本体，严重制约跨本体泛化研究。RoboCOIN 虽含 15 种本体，但每种本体的任务覆盖稀疏。
RoboMIND 2.0 是首个同时支持双臂协调、移动操作、灵巧手与高保真触觉传感的开放数据集。Representative existing datasets are each limited along a single dimension: BridgeData V2 (60K trajectories, only 13 skills), DROID (76K trajectories, single-arm Franka only), Open X-Embodiment (an aggregation of data, no tactile sensing). Even the most recent AgiBot World (1M trajectories) and Galaxea Open-World (50K) rely on a single robot embodiment, which severely constrains research on cross-embodiment generalization. RoboCOIN does cover 15 embodiments, but task coverage per embodiment is sparse. RoboMIND 2.0 is the first open dataset to support bimanual coordination, mobile manipulation, dexterous hands and high-fidelity tactile sensing at the same time.
图 1：RoboMIND 2.0 总览。数据集包含来自六种异构机器人本体（Franka、UR5e、AgileX、ARX、Tien Kung、Tian Yi）
的 310K 双臂轨迹，总时长逾 1,000 小时，含 12K 触觉增强序列与 20K 移动操作轨迹。
所有数据均通过统一的遥操作与质量保障流程采集，并配备精细化自然语言标注。
配套发布的数字孪生 USD 资产与 20K 仿真轨迹支持 sim-to-real 迁移研究。Figure 1: Overview of RoboMIND 2.0. The dataset contains 310K bimanual trajectories from six heterogeneous robot embodiments (Franka, UR5e, AgileX, ARX, Tien Kung, Tian Yi), totalling more than 1,000 hours, including 12K tactile-augmented sequences and 20K mobile manipulation trajectories. All data were collected through a unified teleoperation and quality-assurance pipeline and come with fine-grained natural language annotations. The accompanying digital-twin USD assets and 20K simulation trajectories support sim-to-real transfer research.
310K+双臂操作轨迹bimanual manipulation trajectories
759复杂任务 / 129 项技能complex tasks / 129 skills
6异构机器人本体heterogeneous robot embodiments
1,139操作对象种类categories of manipulated objects
在任务类型上，"Grasp and Place" 占比最高（31.5%，97,423 条轨迹），其次为 "Special"（19.8%）、
"Interactive"（19.3%）；场景覆盖家庭（客厅、厨房、儿童房）与工业（物流分拣、生物实验室、工业装配线）
各约 50%，真正实现了机器人研究所需的多维度多样性。By task type, "Grasp and Place" takes the largest share (31.5%, 97,423 trajectories), followed by "Special" (19.8%) and "Interactive" (19.3%); scenes are split roughly 50% each between household settings (living room, kitchen, children's room) and industrial ones (logistics sorting, biology laboratory, industrial assembly line), truly delivering the multi-dimensional diversity that robotics research requires.
02方法Method
RoboMIND 2.0 在数据层面构建了统一化的多模态采集与质量保障体系；
在模型层面提出 MIND-2——一个通过 offline reinforcement learning 优化的层级化双系统框架，
将高层语义规划与低层精准动作执行有机结合。At the data level, RoboMIND 2.0 builds a unified multimodal collection and quality-assurance system; at the model level, it proposes MIND-2, a hierarchical dual-system framework optimized by offline reinforcement learning that organically combines high-level semantic planning with low-level precise action execution.
数据采集与质量保障Data Collection and Quality Assurance
六种双臂平台各有专属遥操作方案：Franka/UR5e 采用平行布局与 master 手臂映射；
AgileX/ARX 使用 VR 头显实现全身沉浸式操控；Tien Kung/Tian Yi 人形机器人通过外骨骼捕获精细灵巧手动作。
所有轨迹经过 12 类人工数据检查，涵盖完成度、轨迹异常、过快速度、视觉伪影等，
确保本体感受与视觉观测的高度一致性。Each of the six bimanual platforms has a dedicated teleoperation scheme: Franka/UR5e adopt a parallel layout with master-arm mapping; AgileX/ARX use VR headsets for fully immersive whole-body control; the Tien Kung/Tian Yi humanoids capture fine dexterous-hand motions through exoskeletons. Every trajectory passes 12 categories of manual data inspection, covering completion, trajectory anomalies, excessive speed, visual artifacts and more, ensuring tight consistency between proprioception and visual observation.
语言标注体系Language Annotation System
每条轨迹均附有细粒度自然语言描述，支持 language-conditioned policy learning。
标注流程将视频分割为语义子任务片段，通过 VLM 生成初稿后经人工审校，
形成层级化时序指令（全局任务描述 → 子目标序列），
直接服务于 MIND-2-VLM 的任务分解推理。Every trajectory carries a fine-grained natural language description, supporting language-conditioned policy learning. The annotation pipeline segments each video into semantic sub-task clips, drafts descriptions with a VLM and then has them reviewed by human annotators, forming hierarchical temporal instructions (global task description → subgoal sequence) that directly serve the task-decomposition reasoning of MIND-2-VLM.
图：MIND-2 任务定位与 Prompt 设计。MIND-2-VLM（慢速系统）以全局指令为输入，
通过时序定位将长任务分解为可执行子目标（Task Index），再由 MIND-2-VLA（快速系统）
依据当前视觉观测与本体感受生成精准电机动作。
MIND-2-VLA 通过 Implicit Q-Learning (IQL) 进行 offline RL 后训练，
利用 advantage-weighted regression 过滤失败行为、强化成功执行。Figure: MIND-2 task grounding and prompt design. MIND-2-VLM (the slow system) takes the global instruction as input and decomposes the long task into executable subgoals (Task Index) via temporal grounding; MIND-2-VLA (the fast system) then generates precise motor actions from the current visual observation and proprioception. MIND-2-VLA is post-trained with offline RL through Implicit Q-Learning (IQL), using advantage-weighted regression to filter out failed behaviors and reinforce successful executions.
MIND-2-VLM：慢速高层规划器MIND-2-VLM: The Slow High-Level Planner
MIND-2-VLM 是基于云端部署的"机器人大脑"，能够同时协调不同形态的多台机器人。
它接受自然语言指令，将复杂的长时域任务分解为一系列有序的、具体的子目标（grounded subgoals），
并实时判断当前任务阶段，输出对应的 Task Index 供快速系统执行。
其设计允许单一 VLM 控制异构双机器人协作（如 Tian Yi + AgileX 联合完成超市结账场景）。MIND-2-VLM is a cloud-deployed "robot brain" able to coordinate multiple robots of different morphologies at once. It takes natural language instructions, decomposes complex long-horizon tasks into an ordered series of concrete grounded subgoals, judges the current task stage in real time, and emits the corresponding Task Index for the fast system to execute. Its design lets a single VLM control heterogeneous two-robot collaboration (for example, Tian Yi and AgileX jointly completing a supermarket checkout scenario).
MIND-2-VLA：快速低层执行器MIND-2-VLA: The Fast Low-Level Executor
MIND-2-VLA 接收子目标指令、自中心视觉观测与本体感受信息，输出精准、proprioception-aware 的电机动作序列。
训练分为两阶段：首先在全量移动操作数据上进行全规模预训练（fast-slow 架构），
建立跨任务的通用操作先验；其次在协作任务数据上进行 post-training 微调；
最后施以 Implicit Q-Learning (IQL) offline RL——利用成功与失败轨迹的混合训练，
以 advantage-weighted regression 强化优质行为，显著提升鲁棒性与长时域成功率。MIND-2-VLA takes subgoal instructions, egocentric visual observations and proprioceptive information, and outputs precise, proprioception-aware motor action sequences. Training has two stages: first, full-scale pre-training on the complete mobile manipulation data (fast-slow architecture) to establish a general cross-task manipulation prior; second, post-training fine-tuning on collaborative task data; finally, Implicit Q-Learning (IQL) offline RL is applied, training on a mixture of successful and failed trajectories and using advantage-weighted regression to reinforce high-quality behaviors, markedly improving robustness and long-horizon success rate.
数字孪生与仿真数据Digital Twins and Simulation Data
团队开源了所有物理资产的高保真数字孪生（URDF 模型、场景布局、传感器配置），
并发布 20K 条仿真轨迹（Franka 双臂抓夹 + Tien Kung 双臂灵巧手），
任务结构、物体配置与语言指令均与真实数据精确对齐，
为低成本、可扩展的 sim-to-real 迁移研究奠定基础。The team open-sources high-fidelity digital twins of all physical assets (URDF models, scene layouts, sensor configurations) and releases 20K simulation trajectories (Franka bimanual grippers + Tien Kung bimanual dexterous hands), whose task structures, object configurations and language instructions are precisely aligned with the real data, laying a foundation for low-cost, scalable sim-to-real transfer research.
图 7：RoboMIND 2.0 多维度分布。技能数量：覆盖 7 类核心操作技能，"Grasp and Place" 占比最高（31.5%，97,423 条）；
机器人本体分布：Franka 贡献最多（37.9%），其次为 UR5e（28.3%）；
场景分布：含 7 类代表性场景，工业与家庭场景各占约 50%。Figure 7: Multi-dimensional distribution of RoboMIND 2.0. Skill counts: 7 categories of core manipulation skills are covered, with "Grasp and Place" the largest share (31.5%, 97,423 trajectories); embodiment distribution: Franka contributes the most (37.9%), followed by UR5e (28.3%); scene distribution: 7 categories of representative scenes, with industrial and household scenes each about 50%.
03实验Experiments
评测在六种真实机器人本体上进行，涵盖固定基座双臂、移动双臂与人形机器人。
每个任务测试 10 次，以 task success rate 为主要指标。
单任务 imitation learning 基线：ACT、Dense Policy、DP3、UVA；
多任务 VLA 基线：π0、π0.5、HybridVLA、XR-1。Evaluation is carried out on six real robot embodiments, covering fixed-base bimanual arms, mobile bimanual arms and humanoids. Each task is tested 10 times, with task success rate as the primary metric. Single-task imitation learning baselines: ACT, Dense Policy, DP3, UVA; multi-task VLA baselines: π0, π0.5, HybridVLA, XR-1.
图 12 + 13：多平台双臂操作任务可视化。涵盖 Franka、UR5e、AgileX、ARX、Tien Kung、Tian Yi
六种平台的 38 项评测任务，包括杯具传递、毛巾挂载、开关操作、字母分拣等固定基座任务，
以及移动取放、多场景导航操作等移动操作任务。Figures 12 + 13: Visualization of bimanual manipulation tasks across platforms. 38 evaluation tasks on the six platforms Franka, UR5e, AgileX, ARX, Tien Kung and Tian Yi, including fixed-base tasks such as cup handover, towel hanging, switch operation and letter sorting, as well as mobile manipulation tasks such as mobile pick-and-place and multi-scene navigation manipulation.
3D 感知方法（DP3、Dense Policy）在双臂协调任务上整体优于 2D 方法（ACT、UVA），
因其更丰富的空间建模能力能更准确地表示双臂交互的视觉动态。
Dense Policy 在不同机器人本体与任务类型间表现最为一致，适合开放世界多机器人场景；
DP3 在结构化、视觉丰富的固定臂环境中表现突出，但在 AgileX-MV 等移动平台上性能下降明显。3D perception methods (DP3, Dense Policy) outperform 2D methods (ACT, UVA) overall on bimanual coordination tasks, because their richer spatial modeling capacity represents the visual dynamics of bimanual interaction more accurately. Dense Policy is the most consistent across robot embodiments and task types, making it suitable for open-world multi-robot settings; DP3 stands out in structured, visually rich fixed-arm environments, but its performance drops markedly on mobile platforms such as AgileX-MV.
VLA 模型评测VLA Model Evaluation
模型Model
AgileX-MV-Task1
AgileX-MV-Task2
AgileX-MV-Task3
AgileX-MV-Task4
π0
0.1
0.0
0.0
0.1
π0.5
0.3
0.0
0.3
0.1
XR-1
0.4
0.2
0.4
0.3
MIND-2
0.5
0.8
0.4
0.7
π0 因训练以互联网数据为主、缺乏 grounded motor prior，在双臂与移动场景几乎全部失败（近零成功率）。
XR-1 展现出最强的跨本体泛化能力，在固定双臂、移动机械臂与全身人形机器人上均取得较高成功率。
MIND-2 在全部 AgileX 移动操作任务上均超越所有 VLA 基线，最大优势达 +0.6（Task2）。Because π0 is trained mainly on internet data and lacks a grounded motor prior, it fails on almost all bimanual and mobile scenarios (near-zero success rate). XR-1 shows the strongest cross-embodiment generalization, achieving relatively high success rates on fixed bimanual arms, mobile arms and whole-body humanoids. MIND-2 surpasses every VLA baseline on all AgileX mobile manipulation tasks, with a maximum margin of +0.6 (Task2).
在 Tian Yi + AgileX 协同的三个长时域场景（超市结账、工业开关分拣、化学实验室配液）中，
主流 imitation learning 与 VLA 模型均表现不佳。
MIND-2（Offline RL）——施加 IQL 后训练——在全部三项任务上取得最高成功率（0.9 / 0.8 / 0.6），
且当成功与失败轨迹 1:1 混合训练时效果最优（三项分别达 1.0 / 1.0 / 0.8）。Across the three long-horizon scenarios jointly performed by Tian Yi and AgileX (supermarket checkout, industrial switch sorting, chemistry laboratory liquid dispensing), mainstream imitation learning and VLA models all perform poorly. MIND-2 (Offline RL), post-trained with IQL, attains the highest success rate on all three tasks (0.9 / 0.8 / 0.6), and works best when successful and failed trajectories are mixed 1:1 for training (reaching 1.0 / 1.0 / 0.8 respectively).
触觉传感消融Tactile Sensing Ablation
在 AgileX 四项移动操作任务上，将触觉信号融入本体感受输入后：
π0.5 成功率平均提升约 0.1；XR-1 提升更为显著（如 Task1: 0.4 → 0.6，Task2: 0.2 → 0.4）。
结论为："incorporating tactile information consistently improves the success rates across multiple mobile
manipulation tasks, with particularly pronounced gains in scenarios requiring fine manipulation or physical interaction"。On the four AgileX mobile manipulation tasks, fusing tactile signals into the proprioceptive input raises the π0.5 success rate by about 0.1 on average, while XR-1 improves far more markedly (e.g. Task1: 0.4 → 0.6, Task2: 0.2 → 0.4). The conclusion: "incorporating tactile information consistently improves the success rates across multiple mobile manipulation tasks, with particularly pronounced gains in scenarios requiring fine manipulation or physical interaction".
Sim-to-Real 迁移与仿真 BenchmarkSim-to-Real Transfer and the Simulation Benchmark
在 Tien Kung 仿真任务中，XR-1 在四项任务上分别达 48/50、39/50、46/50、31/50，
验证了仿真数据的高保真度。混合真实与仿真数据训练时，所有模型均受益；
进一步将仿真数据比例从 1:1 提升至 1:5 仍可改善真实机器人性能，
XR-1 在 Task3 上从 0.8 提升至 0.9，Task4 从 0.5 提升至 0.7。On the Tien Kung simulation tasks, XR-1 reaches 48/50, 39/50, 46/50 and 31/50 on the four tasks respectively, confirming the high fidelity of the simulation data. All models benefit from training on a mixture of real and simulated data; further raising the simulation-data ratio from 1:1 to 1:5 still improves real-robot performance, with XR-1 going from 0.8 to 0.9 on Task3 and from 0.5 to 0.7 on Task4.
04局限性Limitations
说明：本文 Discussion 章节明确讨论了未来扩展方向，以下局限性综合论文明确陈述（标注"stated"）
与设计层面推断（标注"inferred"）。Note: The paper's Discussion section explicitly addresses future extension directions; the limitations below combine what the paper states explicitly (marked "stated") with design-level inferences (marked "inferred").
单一触觉传感器类型（stated）A single type of tactile sensor (stated)
当前触觉数据仅源自特定型号的压力传感器，未涵盖力矩传感器、音频等其他物理交互信号。
论文明确指出将在后续工作中扩展更多模态（"e.g., force-torque and audio"）。The current tactile data come only from one specific model of pressure sensor and do not cover other physical interaction signals such as force-torque sensors or audio. The paper explicitly notes that more modalities will be added in follow-up work ("e.g., force-torque and audio").
尽管已覆盖六种双臂平台，但论文明确表示会"continue expanding RoboMIND 2.0 with new embodiments, skills, and modalities"。
部分稀有本体（如 ARX、Tien Kung）的任务数量相对有限，可能制约跨本体泛化研究的深度（inferred）。Although six bimanual platforms are already covered, the paper explicitly states that it will "continue expanding RoboMIND 2.0 with new embodiments, skills, and modalities". The task counts for some rarer embodiments (such as ARX and Tien Kung) are relatively limited, which may constrain the depth of cross-embodiment generalization research (inferred).
MIND-2 对大规模预训练数据的强依赖（inferred）MIND-2's strong dependence on large-scale pre-training data (inferred)
MIND-2（Post Training）直接微调现有 VLA 模型（InternVL3 + π0.5）在协作任务上成功率偏低（0.6/0.6/0.4），
而 Full-scale Training 版本需要在全量移动操作数据上预训练，计算成本显著更高。
对于计算资源受限的研究者，低成本快速部署能力仍存在挑战（inferred）。MIND-2 (Post Training), which directly fine-tunes existing VLA models (InternVL3 + π0.5), reaches only low success rates on the collaboration tasks (0.6/0.6/0.4), whereas the Full-scale Training variant must be pre-trained on the complete mobile manipulation data at markedly higher compute cost. For researchers with limited compute resources, low-cost rapid deployment remains a challenge (inferred).
化学实验室等复杂序列任务的上限（stated）The ceiling on complex sequential tasks such as the chemistry laboratory (stated)
即便是 MIND-2（Offline RL）在最复杂的化学实验室协作任务中成功率仅为 0.6，
说明长序列、强时序依赖的多机器人任务仍是当前方法的重要瓶颈（inferred from Table 4 results）。Even MIND-2 (Offline RL) reaches a success rate of only 0.6 on the most complex chemistry laboratory collaboration task, showing that long-sequence, strongly time-dependent multi-robot tasks remain a major bottleneck for current methods (inferred from Table 4 results).
