GM-100 提出 100 个精细设计的机器人操作任务，通过系统分析现有数据集的任务偏差，结合人体-物体交互原语（HAKE）和大语言模型扩展，构建覆盖长尾行为的多样化评估基准，用以区分不同 VLA 模型的真实能力。GM-100 introduces 100 detail-oriented robot manipulation tasks. By systematically analyzing the task bias of existing datasets and combining human-object interaction primitives (HAKE) with large language model expansion, it builds a diverse evaluation benchmark covering long-tail behaviors, in order to differentiate the true capabilities of different VLA models.
当前机器人学习数据集的任务设计缺乏系统性原则，大量工作集中在少数常见动作（如"pick and hold"），无法有效区分和衡量不同方法的真实能力。Task design in current robot learning datasets lacks systematic principles: a large body of work concentrates on a few common actions (such as "pick and hold"), which fails to effectively distinguish and measure the true capabilities of different methods.
"Do the current datasets and task designs truly advance the capabilities of robotic agents? Do evaluations on a few common tasks accurately reflect the differentiated performance of various methods proposed by different teams and evaluated on different tasks?"
图 1：对现有机器人数据集的词云与动词频率分析。结果显示，现有任务设计高度集中在"pick"、"place"等少数常见动作，长尾行为严重匮乏，导致评估存在显著偏差。Figure 1: Word cloud and verb frequency analysis of existing robot datasets. The results show that existing task designs are highly concentrated on a few common actions such as "pick" and "place", while long-tail behaviors are severely lacking, which introduces significant bias into evaluation.
100精细设计任务数Detail-oriented tasks
13K+收集轨迹总数Total collected trajectories
2真实机器人平台Real robot platforms
3基线 VLA 模型对比Baseline VLA models compared
现有数据集存在"重叠过多、设计随意"的问题——不同团队在各自不同的任务上评估自己的方法，缺乏统一的多样化测试集，难以做到横向对比。GM-100 的目标是成为机器人学习领域的"Olympics"：提供标准化、多样化、且具有足够难度的任务集合，让不同方法在同一赛场上同台竞技。Existing datasets suffer from "excessive overlap and arbitrary design" — different teams evaluate their own methods on their own different tasks, and the absence of a unified, diverse test set makes horizontal comparison difficult. The goal of GM-100 is to become the "Olympics" of robot learning: to provide a standardized, diverse and sufficiently difficult set of tasks, so that different methods compete on the same field.
02方法Method
GM-100 的任务构建遵循"分析现有任务→LLM 语义扩展→混合过滤→专家筛选"的完整流水线，结合人体-物体交互原语数据库（HAKE、OCL）引入丰富的长尾行为。The task construction of GM-100 follows a complete pipeline of "analyzing existing tasks → LLM semantic expansion → hybrid filtration → expert selection", and draws on human-object interaction primitive databases (HAKE, OCL) to introduce rich long-tail behaviors.
图 2："The construction pipeline of the GM-100 benchmark. The process begins with collecting existing robot tasks, followed by a semantic expansion using HAKE and LLM-based generation to cover long-tail interactions. The candidates then undergo a rigorous hybrid filtration by LLMs and human experts to ensure hardware feasibility and data collection friendliness. Finally, 100 high-priority tasks are selected and instantiated with detailed interaction criteria and template videos."（原文引用）Figure 2: "The construction pipeline of the GM-100 benchmark. The process begins with collecting existing robot tasks, followed by a semantic expansion using HAKE and LLM-based generation to cover long-tail interactions. The candidates then undergo a rigorous hybrid filtration by LLMs and human experts to ensure hardware feasibility and data collection friendliness. Finally, 100 high-priority tasks are selected and instantiated with detailed interaction criteria and template videos." (quoted from the paper)
步骤一：现有任务分析与去重Step 1: Analysis and Deduplication of Existing Tasks
收集 Agibot 和 π₀.₅ 等公开数据集中的现有机器人任务，去除重复项并进行语义归类，通过词云和动词频率分布可视化任务偏差，识别哪些交互类型在现有数据集中过度代表或严重缺失。Existing robot tasks are collected from public datasets such as Agibot and π₀.₅, duplicates are removed and the tasks are grouped semantically; word clouds and verb frequency distributions visualize the task bias and identify which interaction types are over-represented or severely missing in existing datasets.
步骤二：LLM 驱动的任务扩展Step 2: LLM-driven Task Expansion
以 Qwen3 模型为基础，设计精心构造的 prompt，融入来自 HAKE 和 OCL 数据库的人体-物体交互原语（human-object interaction primitives）与物体功能可供性（object affordances），生成候选任务列表，重点覆盖长尾行为——即现有数据集中出现频率极低的复杂操作。Based on the Qwen3 model, carefully constructed prompts are designed that incorporate human-object interaction primitives and object affordances from the HAKE and OCL databases, generating a list of candidate tasks that focuses on long-tail behaviors — that is, complex manipulations occurring extremely rarely in existing datasets.
步骤三：混合过滤与专家验证Step 3: Hybrid Filtration and Expert Verification
候选任务经过三级过滤：(1) 词义消歧（word sense disambiguation）去除歧义任务；(2) LLM 自动评分——对每个任务的硬件可行性和数据采集友好性进行打分；(3) 5 位人类专家进行人工验证，最终按综合得分优先级选出 100 个任务，并为每个任务配备详细的交互标准说明和模板视频。Candidate tasks go through three levels of filtration: (1) word sense disambiguation removes ambiguous tasks; (2) automatic LLM scoring — rating the hardware feasibility and data collection friendliness of every task; (3) manual verification by 5 human experts. Finally, 100 tasks are selected by priority of their overall score, and each task is equipped with detailed interaction criteria and a template video.
数据采集协议Data Collection Protocol
每个任务收集 100 条带有不同初始条件和扰动设计（varying initial conditions and design perturbations）的轨迹，确保位置、朝向和物体摆放的多样性；另外再采集 30 条分布相近的测试轨迹。前 10 个任务在两个平台上各采集 130 条轨迹；任务 11–100 仅在 Cobot Magic 平台上采集。全部数据共超过 13,000 条轨迹。For every task, 100 trajectories with varying initial conditions and design perturbations are collected, ensuring diversity of position, orientation and object placement; a further 30 test trajectories of a similar distribution are collected. The first 10 tasks are collected on both platforms, 130 trajectories each; tasks 11–100 are collected only on the Cobot Magic platform. In total the data comprises more than 13,000 trajectories.
图 3：GM-100 使用的两款真实机器人平台。左：Agilex Cobot Magic——类 Mobile-Aloha 结构，前伸式机械臂，头部安装相机；右：Dobot Xtrainer——类 Aloha 结构，内折式机械臂，俯视相机视角。两个平台的结构差异可验证评估方法的平台泛化性。Figure 3: The two real robot platforms used by GM-100. Left: Agilex Cobot Magic — a Mobile-Aloha-like structure with forward-extending arms and a head-mounted camera; right: Dobot Xtrainer — an Aloha-like structure with inward-folding arms and a top-down camera view. The structural difference between the two platforms allows the platform generalization of the evaluation to be verified.
03实验Experiments
在两个真实机器人平台上对三个基线模型（DP、π₀、π₀.₅）进行评估，使用 Partial Success Rate（PSR）和 Success Rate（SR）两个指标衡量性能。Three baseline models (DP, π₀, π₀.₅) are evaluated on two real robot platforms, and performance is measured with two metrics, Partial Success Rate (PSR) and Success Rate (SR).
图 4："Partial Success Rate on Cobot Magic Platform. The color intensity in the heatmap indicates the PSR."（原文引用）热力图展示了三个基线模型在 100 个任务上的 PSR 分布，颜色越深表示成功率越高。可以看到绝大多数任务对所有基线模型都极具挑战性。Figure 4: "Partial Success Rate on Cobot Magic Platform. The color intensity in the heatmap indicates the PSR." (quoted from the paper) The heatmap shows the PSR distribution of the three baseline models over the 100 tasks; darker colors indicate higher success rates. The vast majority of tasks turn out to be extremely challenging for all baseline models.图 5："Task-level diverging comparison of Normalized MSE (left) and Partial Success Rate (right) across models."（原文引用）MSE 与物理成功率之间存在反向相关关系：动作预测误差更低的模型通常能实现更高的 PSR，验证了 GM-100 评估指标的一致性。Figure 5: "Task-level diverging comparison of Normalized MSE (left) and Partial Success Rate (right) across models." (quoted from the paper) An inverse correlation exists between MSE and physical success rate: models with lower action prediction error generally achieve higher PSR, which confirms the consistency of the GM-100 evaluation metrics.
关键发现Key Findings
任务可行性验证：实验表明 GM-100 的 100 个任务对机器人系统而言均具备可执行性（feasible to execute）。Task feasibility verified: Experiments show that all 100 tasks of GM-100 are feasible to execute for robot systems.
模型区分能力：不同任务上三个基线的 PSR 差异显著（如 Task 0007：DP 6.2% vs π₀.₅ 90.0%），证明 GM-100 能有效区分不同 VLA 模型的能力。Model discrimination: The PSR of the three baselines differs markedly across tasks (e.g. Task 0007: DP 6.2% vs π₀.₅ 90.0%), showing that GM-100 effectively distinguishes the capabilities of different VLA models.
总体挑战性高：即便是最强基线 π₀.₅，其平均 SR 也仅为 24.9%，平均 PSR 为 53.9%，说明当前 VLA 模型距离熟练完成多样化操作任务仍有较大差距。High overall difficulty: Even the strongest baseline π₀.₅ attains an average SR of only 24.9% and an average PSR of 53.9%, indicating that current VLA models are still far from proficiently completing diverse manipulation tasks.
MSE 与成功率正相关：动作预测误差（MSE/L1）与物理成功率（PSR/SR）之间呈现一致的反向相关关系，验证了预测误差作为代理指标的合理性。MSE correlates with success rate: Action prediction error (MSE/L1) and physical success rate (PSR/SR) exhibit a consistent inverse correlation, which validates prediction error as a proxy metric.
Task 0005 特例：DP 在该任务上 PSR/SR 均为 12.2%，略优于 π₀.₅ 的 8.2%，说明基准中存在对不同模型特性敏感的多样化任务。The Task 0005 exception: On this task DP attains 12.2% for both PSR and SR, slightly better than the 8.2% of π₀.₅, showing that the benchmark contains diverse tasks sensitive to different model characteristics.
04局限性Limitations
说明：本文无独立的 Limitations 章节；以下内容部分为作者在论文中明确陈述（标注"原文"），部分为从设计层面推断（标注"推断"）。Note: The paper has no dedicated Limitations section; part of the content below is explicitly stated by the authors in the paper (marked "from the paper"), and part is inferred at the design level (marked "inferred").
整体成功率偏低（原文）Low overall success rate (from the paper)
"Suboptimal robotic arm configurations for specific tasks, the wide distribution of collected datasets, and insufficient training data collectively contribute to low overall success rates on GM-100 benchmarks."（原文）机械臂配置欠优、数据分布宽泛、训练数据不足是当前成功率偏低的主要原因。这也意味着 GM-100 的完整解决仍需大量方法和数据上的改进。"Suboptimal robotic arm configurations for specific tasks, the wide distribution of collected datasets, and insufficient training data collectively contribute to low overall success rates on GM-100 benchmarks." (from the paper) Suboptimal robotic arm configurations, broad data distributions and insufficient training data are the main reasons for the currently low success rates. This also means that fully solving GM-100 still requires substantial improvements in both methods and data.
测试环境的公平性局限（原文）Fairness limitations of the testing environment (from the paper)
"We do not aim to build an absolutely fair physical testing environment, as current robotic learning models remain significantly influenced by tester capability and environmental conditions."（原文）真实机器人测试高度依赖操作员水平和环境条件，难以做到完全受控的公平对比；"Real-world robot testing is highly costly"也限制了更大规模的实验。"We do not aim to build an absolutely fair physical testing environment, as current robotic learning models remain significantly influenced by tester capability and environmental conditions." (from the paper) Real robot testing depends heavily on operator skill and environmental conditions, so a fully controlled, fair comparison is hard to achieve; "Real-world robot testing is highly costly" also limits larger-scale experiments.
任务覆盖范围尚为初步版本（推断）Task coverage is still a preliminary version (inferred)
作者明确指出 GM-100 是"the first step towards a robot learning Olympics"，当前 100 个任务仅代表第一版（v1）。未来版本可能扩展更多类别、引入更高难度的组合操作或需要工具使用的场景。任务 11–100 的真实环境评估结果尚未公开（受测试成本限制）。The authors explicitly state that GM-100 is "the first step towards a robot learning Olympics", and the current 100 tasks represent only the first version (v1). Future versions may expand to more categories and introduce harder compositional manipulations or scenarios requiring tool use. Real-world evaluation results for tasks 11–100 have not yet been released (limited by testing cost).
平台泛化性有待验证（推断）Platform generalization remains to be verified (inferred)
完整的 100 个任务评估仅在 Agilex Cobot Magic 平台上进行，Dobot Xtrainer 只测试了前 10 个任务。不同平台（如单臂、人形机器人等）上的泛化性能尚未全面评估，限制了基准对更广泛硬件生态的代表性。The full 100-task evaluation was carried out only on the Agilex Cobot Magic platform, while Dobot Xtrainer was tested on the first 10 tasks only. Generalization performance on other platforms (single-arm robots, humanoids, etc.) has not been comprehensively evaluated, which limits how representative the benchmark is of the broader hardware ecosystem.
