可扩展双臂机器人操作数据生成器与强域随机化基准A Scalable Data Generator and Benchmark with Strong Domain Randomization for Bimanual Robotic Manipulation
Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu 等 · Shanghai AI Lab · HKU MMLab · SJTU · THUTianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu et al. · Shanghai AI Lab · HKU MMLab · SJTU · THU
RoboTwin 2.0 是一个面向双臂机器人操作的大规模合成数据生成框架，集成了基于多模态大语言模型（MLLM）的闭环专家代码生成、五维域随机化以及机器人本体感知自适应抓取。该系统包含 731 个物体组成的操作数据集，覆盖 147 个类别，预生成超过 10 万条专家轨迹，并提供 50 任务双臂操作基准，显著提升了策略在真实世界中的泛化能力。RoboTwin 2.0 is a large-scale synthetic data generation framework for bimanual robotic manipulation. It integrates closed-loop expert code generation driven by a multimodal large language model (MLLM), five-dimensional domain randomization, and embodiment-aware adaptive grasping. The system ships a manipulation object dataset of 731 objects spanning 147 categories, pre-generates more than 100000 expert trajectories, and provides a 50-task bimanual manipulation benchmark, markedly improving the real-world generalization of learned policies.
双臂机器人操作bimanual robotic manipulation合成数据生成synthetic data generation域随机化domain randomizationsim-to-realbimanual manipulationMLLM 代码生成MLLM code generationembodiment-awarebenchmark
01动机Motivation
双臂机器人操作需要海量高质量轨迹数据，而现有合成数据管线在自动质量控制、场景多样性和跨机器人平台泛化三方面存在根本缺陷。Bimanual robotic manipulation requires massive amounts of high-quality trajectory data, yet existing synthetic data pipelines are fundamentally deficient in three respects: automated quality control, scene diversity, and generalization across robot platforms.
"lack of automated quality control: without an expert-level validation loop, many generated trajectories include execution failures or suboptimal grasps; domain randomization is often superficial, yielding overly clean and homogeneous scenes; overlooking cross-embodiment variation: different bimanual platforms can differ substantially in their kinematic capabilities."
图 1：RoboTwin 2.0 系统概览。 框架包含：（左）基于 MLLM 的专家数据生成管线；（中）覆盖 50 个任务的双臂操作基准，支持 Franka、Piper、UR5、ARX-X5、Aloha-AgileX 五种机器人平台；（右）包含 731 个物体、147 个类别的 RoboTwin Object Dataset（RoboTwin-OD），每个物体均标注了操作姿态候选。Figure 1: Overview of the RoboTwin 2.0 system. The framework comprises: (left) an MLLM-based expert data generation pipeline; (middle) a bimanual manipulation benchmark covering 50 tasks and supporting five robot platforms — Franka, Piper, UR5, ARX-X5 and Aloha-AgileX; (right) the RoboTwin Object Dataset (RoboTwin-OD) with 731 objects across 147 categories, each annotated with manipulation pose candidates.
50双臂操作任务基准bimanual manipulation tasks in the benchmark
5支持的机器人平台supported robot platforms
367%10 条真实样本相对增益（vs. 纯真实数据基线）relative gain with 10 real samples (vs. real-data-only baseline)
真实数据采集成本高昂且难以规模化，而直接使用合成数据训练的策略往往因 sim-to-real 差距而失败。RoboTwin 2.0 通过三个相互协同的组件来解决上述问题：自动化专家数据生成、全面的域随机化，以及机器人本体感知自适应，从而生成 "high-quality, diverse, realistic, and interaction-rich datasets for bimanual manipulation"。Real-world data collection is expensive and hard to scale, while policies trained directly on synthetic data often fail because of the sim-to-real gap. RoboTwin 2.0 addresses these issues through three mutually reinforcing components: automated expert data generation, comprehensive domain randomization, and embodiment-aware adaptation, so as to produce "high-quality, diverse, realistic, and interaction-rich datasets for bimanual manipulation".
02方法Method
RoboTwin 2.0 由三大核心组件构成：MLLM 驱动的闭环专家代码生成、五维系统性域随机化，以及基于物体 affordance 的机器人本体感知抓取适配。RoboTwin 2.0 is built from three core components: MLLM-driven closed-loop expert code generation, systematic five-dimensional domain randomization, and embodiment-aware grasp adaptation based on object affordance.
图 2：RoboTwin 2.0 数据生成管线。 给定任务描述，MLLM 生成 Python 任务程序，在仿真中执行并由视觉-语言观察者模型进行逐帧诊断，根据"multimodal execution feedback"迭代修正代码，直到程序达到目标成功率或耗尽最大修正轮次（5 次）。合格程序随后叠加五维域随机化，生成多样化的专家轨迹。Figure 2: The RoboTwin 2.0 data generation pipeline. Given a task description, the MLLM produces a Python task program that is executed in simulation and diagnosed frame by frame by a vision-language observer model; the code is iteratively revised according to "multimodal execution feedback" until the program reaches the target success rate or exhausts the maximum number of revision rounds (5). Qualified programs are then overlaid with five-dimensional domain randomization to generate diverse expert trajectories.
系统采用闭环架构，由代码生成模型与视觉-语言观察者模型协同工作。每次生成的任务程序在仿真中执行 10 次以评估随机性影响。观察者模型对失败执行进行逐帧检查，诊断失败模式（如抓取轴错误、碰撞干涉等），并向代码生成模型反馈精准的错误定位，指导代码修复。最多允许 5 次连续迭代修正，未达标则终止并标记为失败任务。与仅使用文本反馈相比，多模态反馈将 Average Success Rate (ASR) 从 62.1% 提升至 71.3%，同时将 token 消耗从 1236.6 降至 839.7。The system adopts a closed-loop architecture in which a code generation model and a vision-language observer model work in concert. Each generated task program is executed 10 times in simulation to assess the effect of randomness. The observer model inspects failed executions frame by frame, diagnoses failure modes (wrong grasp axis, collision interference, and so on), and returns precise error localization to the code generation model to guide repair. At most 5 consecutive iterative revisions are allowed; a program that still falls short is terminated and marked as a failed task. Compared with text-only feedback, multimodal feedback raises the Average Success Rate (ASR) from 62.1% to 71.3% while reducing token cost from 1236.6 to 839.7.
图 3：专家代码生成闭环架构。 代码生成代理（code-generation agent）与视觉-语言观察者代理（VLM observer agent）交替工作，观察者对仿真执行视频进行帧级检查，提供故障定位诊断，驱动代码迭代修复，直至达到目标成功率。Figure 3: The closed-loop architecture of expert code generation. A code-generation agent and a VLM observer agent work in alternation; the observer performs frame-level inspection of the simulated execution video, provides fault-localization diagnoses, and drives iterative code repair until the target success rate is reached.
针对场景外观和语言指令实施五个维度的系统化增广，从根本上解决仿真到现实的视觉差距：Systematic augmentation is applied along five dimensions of scene appearance and language instruction, addressing the visual sim-to-real gap at its root:
杂乱物体（Clutter）：在场景中随机放置干扰物，模拟真实工作台的混乱状态Clutter: distractor objects are randomly placed in the scene to mimic the messy state of a real workbench
背景纹理（Background Textures）：包含 11,000 张精选纹理，覆盖多样化桌面和环境背景Background Textures: 11,000 curated textures covering diverse tabletops and environment backgrounds
光照变化（Lighting Variation）：随机化光源位置、强度和色温Lighting Variation: light source position, intensity and color temperature are randomized
桌面高度（Tabletop Heights）：模拟不同高度的操作台Tabletop Heights: workbenches of differing height are simulated
语言指令多样化（Language Instructions）：组合增广生成大量语言变体，丰富策略的语言条件输入Language Instructions: combinatorial augmentation produces a large number of linguistic variants, enriching the language-conditioned input of the policy
图 4：五维域随机化可视化与纹理库。 从左到右展示了杂乱物体、背景纹理（来自 11,000 张纹理库）、光照变化、桌面高度、语言指令五个维度的随机化效果，以及典型任务在不同域随机化配置下的场景对比。Figure 4: Visualization of the five-dimensional domain randomization and the texture library. From left to right, the randomization effects of the five dimensions — clutter, background texture (drawn from the library of 11,000 textures), lighting variation, tabletop height and language instruction — together with scene comparisons of a representative task under different domain randomization configurations.
不同双臂平台在运动学上差异显著（如 Piper 自由度受限，Aloha-AgileX 臂展不同），通用抓取姿态往往导致部分平台无法执行。RoboTwin 2.0 为 RoboTwin-OD 中的每个物体标注多个候选操作姿态（覆盖不同抓取轴），通过角度扰动偏向更高可达性的抓取候选，实现平台专属的抓取执行。该策略在全平台平均提升 8.3% 的操作成功率，其中低自由度平台 Piper 提升最高达 +22.7%，Aloha-AgileX 提升 +13.7%。Different bimanual platforms differ substantially in kinematics (Piper, for instance, has limited degrees of freedom, and Aloha-AgileX has a different arm reach), so a universal grasp pose often leaves some platforms unable to execute the task. RoboTwin 2.0 annotates every object in RoboTwin-OD with multiple candidate manipulation poses (covering different grasp axes) and, through angular perturbation, biases selection toward grasp candidates with higher reachability, achieving platform-specific grasp execution. This strategy raises the manipulation success rate by 8.3% on average across all platforms, with the largest gain of +22.7% on the low-DoF Piper platform and +13.7% on Aloha-AgileX.
图 7：RoboTwin Object Dataset（RoboTwin-OD）。 包含 147 个类别、731 个带操作姿态标注的三维物体模型。每个物体标注了多个候选抓取轴，用于机器人本体感知的自适应抓取选择。Figure 7: The RoboTwin Object Dataset (RoboTwin-OD). It contains 731 3D object models with manipulation pose annotations across 147 categories. Each object is annotated with multiple candidate grasp axes, used for embodiment-aware adaptive grasp selection.
03实验Experiments
在代码生成质量、策略鲁棒性（仿真）以及 sim-to-real 迁移三个维度上进行全面评估，涵盖 RDT、Pi0、DP3 等主流策略学习方法和 5 种机器人平台。A comprehensive evaluation is carried out along three dimensions — code generation quality, policy robustness in simulation, and sim-to-real transfer — covering mainstream policy learning methods such as RDT, Pi0 and DP3 as well as 5 robot platforms.
策略鲁棒性：域随机化数据的影响（仿真）Policy Robustness: Effect of Domain-Randomized Data (Simulation)
在 50 任务基准（Easy vs. Hard 配置）上评估 RDT、Pi0、DP3 等策略。使用域随机化数据训练相比 clean 数据带来显著提升，且 Hard 配置下分数骤降凸显了真实鲁棒性挑战：Policies such as RDT, Pi0 and DP3 are evaluated on the 50-task benchmark (Easy vs. Hard configurations). Training on domain-randomized data brings a marked gain over clean data, while the steep score drop under the Hard configuration highlights the genuine robustness challenge:
在 Franka 平台上进行四种配置（seen/unseen 场景 × clean/cluttered 环境）的真实实验，使用 10 条真实演示 + 1000 条合成轨迹的 few-shot 设置：Real-world experiments are conducted on the Franka platform under four configurations (seen/unseen scene × clean/cluttered environment), using a few-shot setting of 10 real demonstrations plus 1000 synthetic trajectories:
配置Configuration
Seen Clean
Seen Cluttered
Unseen Clean
Unseen Cluttered
域随机化合成数据增益gain from domain-randomized synthetic data
+13.5%
+27.5%
+23.5%
+33.0%
在 unseen 背景的 zero-shot 配置（仅合成数据，无真实演示）下，成功率分别提升 +21.0% 和 +20.5%。总体而言，10 条真实样本 + 合成数据相对于仅使用真实数据的基线实现了 367% 的相对增益。Under the zero-shot configuration with unseen backgrounds (synthetic data only, no real demonstrations), success rates improve by +21.0% and +20.5% respectively. Overall, 10 real samples plus synthetic data deliver a 367% relative gain over the real-data-only baseline.
图 10：真实世界评估（四种测试配置）。 从左到右依次为：seen clean、seen cluttered、unseen clean、unseen cluttered 四种场景配置下的机器人执行关键帧，覆盖带/不带杂乱物体与已见/未见背景的组合，系统性验证了域随机化在真实部署中的有效性。Figure 10: Real-world evaluation (four test configurations). From left to right: key frames of robot execution under the four scene configurations seen clean, seen cluttered, unseen clean and unseen cluttered, covering combinations of with/without clutter and seen/unseen backgrounds, systematically validating the effectiveness of domain randomization in real deployment.图 8：50 个双臂操作任务关键帧总览。 涵盖从简单单步抓放到复杂多步组装、工具使用等各类任务，任务设计兼顾 Easy 和 Hard 难度配置，用于全面评估策略的操作能力。Figure 8: Overview of key frames from the 50 bimanual manipulation tasks. They range from simple single-step pick-and-place to complex multi-step assembly and tool use; the task design covers both Easy and Hard difficulty configurations and serves to comprehensively evaluate the manipulation capability of a policy.
消融分析Ablation Analysis
对域随机化各维度的消融结果表明，杂乱物体和背景纹理对 sim-to-real 性能贡献最大；Clean 数据训练对域随机化测试场景的提升"showed negligible improvements"，说明五维随机化不可或缺。机器人本体感知抓取适配的消融同样显示，低自由度平台（如 Piper）获益最明显（+22.7%），验证了平台专属抓取候选策略的必要性。Ablations over the individual dimensions of domain randomization show that clutter and background texture contribute the most to sim-to-real performance; training on clean data "showed negligible improvements" on domain-randomized test scenes, which indicates that the five-dimensional randomization is indispensable. The ablation of embodiment-aware grasp adaptation likewise shows that low-DoF platforms such as Piper benefit the most (+22.7%), confirming the necessity of platform-specific grasp candidates.
04局限性Limitations
注： 论文未设独立 Limitations 章节。以下第一条为论文正文中明确陈述（stated），其余为从系统设计中推断（inferred）。Note: The paper has no standalone Limitations section. The first item below is explicitly stated in the paper; the others are inferred from the system design.
视觉观察者诊断准确率有限（stated）Limited diagnostic accuracy of the vision-based observer (stated)
论文指出，视觉-语言观察者模型对失败执行的错误定位准确率仅约 30%。"failures from invisible factors like incorrect grasp axis parameters remain challenging for purely vision-based observers to diagnose."这意味着部分失败轨迹可能无法被有效过滤，影响数据质量的上限。The paper notes that the vision-language observer model localizes errors in failed executions with an accuracy of only about 30%. "failures from invisible factors like incorrect grasp axis parameters remain challenging for purely vision-based observers to diagnose." This means that some failed trajectories cannot be filtered out effectively, capping the attainable data quality.
Hard 配置下策略性能大幅下降（stated/inferred）Policy performance drops sharply under the Hard configuration (stated/inferred)
即便使用域随机化数据，策略在 Hard 基准配置下的成功率相较 Easy 配置仍显著下降（DP3 从 55.2% 跌至 5.0%，Pi0 从 46.4% 跌至 16.3%），说明现有策略对强随机化场景的鲁棒性仍有 "significant room for improvement"。Even with domain-randomized data, success rates under the Hard benchmark configuration still fall markedly relative to Easy (DP3 drops from 55.2% to 5.0%, Pi0 from 46.4% to 16.3%), showing that the robustness of existing policies to strongly randomized scenes still leaves "significant room for improvement".
数据生成规模与任务多样性的扩展边界（inferred）Scaling boundary of data generation volume and task diversity (inferred)
当前系统覆盖 50 个任务和 5 种平台，代码生成的 ASR 上限为 71.3%（Top5-ASR 78.6%），约有 20-30% 的任务程序生成失败。对于需要更复杂接触物理或长时程规划的任务，MLLM 驱动的代码生成可靠性可能进一步下降，限制了系统向更广泛操作场景的自动扩展。The current system covers 50 tasks and 5 platforms, and the ceiling of code generation ASR is 71.3% (Top5-ASR 78.6%), meaning that roughly 20-30% of task programs fail to be generated. For tasks that require more complex contact physics or long-horizon planning, the reliability of MLLM-driven code generation may degrade further, limiting automatic extension of the system to broader manipulation scenarios.
真实平台评估范围有限（inferred）Limited scope of real-platform evaluation (inferred)
真实世界实验主要在 Franka 平台上进行，其余四种平台（Piper、UR5、ARX-X5、Aloha-AgileX）的 sim-to-real 结果以仿真为主，真实部署验证尚不完整，不同平台间的迁移效果有待进一步实验确认。Real-world experiments are conducted mainly on the Franka platform; the sim-to-real results for the remaining four platforms (Piper, UR5, ARX-X5, Aloha-AgileX) rest largely on simulation, real deployment validation is still incomplete, and transfer across platforms awaits further experimental confirmation.
