Qwen-RobotManip：对齐解锁机器人操作基础模型的规模化Qwen-RobotManip: Alignment Unlocks Scale for Robotic Manipulation Foundation Models
Alignment Unlocks Scale for Robotic Manipulation Foundation Models
Qwen Team · 2026年6月Qwen Team · June 2026
现有 VLA 模型在标准基准上表现良好，但在真正 OOD 场景下严重退化——根本原因在于异构机器人数据无法被直接对齐后统一缩放。Qwen-RobotManip 以「先对齐、后扩展」为核心原则，在表示、运动、行为三个维度引入统一对齐框架，基于 ~38,100 小时全开源数据构建可泛化 VLA 基础模型，在多项 OOD 基准上大幅超越 π₀.₅ 等当前最优方法。Existing VLA models perform well on standard benchmarks but degrade severely under genuinely OOD conditions—the root cause is that heterogeneous robot data cannot be directly aligned and then scaled up in a unified way. Qwen-RobotManip adopts "align first, then scale" as its core principle, introducing a unified alignment framework along the representation, motion and behavior dimensions, and builds a generalizable VLA foundation model on ~38,100 hours of fully open-source data, substantially surpassing current state-of-the-art methods such as π₀.₅ on multiple OOD benchmarks.
VLA机器人操作Robot manipulationcross-embodiment alignment人机数据合成Human-to-robot data synthesisflow matchingDiffusion Transformer迁移泛化Transfer generalization指令跟随Instruction followingOOD evaluation
01动机Motivation
语言和多模态领域的基础模型因「异构数据来源可在统一公式下对齐，且互联网大量低成本数据允许多样化训练信号相互强化」而实现了强泛化。但机器人操作数据天然异构、昂贵且多样性有限，使得对齐与规模化难以同时实现。Foundation models in language and multimodal domains achieve strong generalization because "heterogeneous data sources can be aligned under a unified formulation, and the vast amount of low-cost internet data lets diverse training signals reinforce one another". Robotic manipulation data, however, is inherently heterogeneous, expensive and limited in diversity, which makes alignment and scaling hard to achieve at the same time.
「We investigate whether this scaling recipe can be applied to robotic manipulation to achieve genuine generalization.」——对齐是数据规模化的先决条件，而非独立的工程选择。"We investigate whether this scaling recipe can be applied to robotic manipulation to achieve genuine generalization." — Alignment is a prerequisite for data scaling, not an independent engineering choice.
图1 系统总览（论文首页图）。
左侧：Human-to-Robot 合成流水线将来自 ~1,933 小时自我中心视频的人手演示渲染为 15 种双臂机器人形态（约 24,808 小时合成演示），再与多源真实机器人数据合并，共构成 >38,100 小时预训练语料。
中间：统一跨形态对齐——表示对齐（规范状态向量）、运动对齐（相机坐标系 EEF delta 姿态）、行为对齐（系统提示 + 上下文自适应）。
右侧：OOD 基准雷达图与数据规模化曲线，显示 Qwen-RobotManip 超越 π₀.₅ 等先前 SOTA，且随数据量增加涌现出 OOD 泛化能力。Figure 1 System overview (paper teaser figure).
Left: the Human-to-Robot synthesis pipeline renders hand demonstrations from ~1,933 hours of egocentric video into 15 dual-arm robot embodiments (about 24,808 hours of synthetic demonstrations), which are then merged with multi-source real-robot data to form a >38,100-hour pre-training corpus.
Middle: unified cross-embodiment alignment—representation alignment (canonical state vector), motion alignment (camera-frame EEF delta pose) and behavior alignment (system prompt + in-context adaptation).
Right: OOD benchmark radar charts and data-scaling curves, showing that Qwen-RobotManip surpasses prior SOTA such as π₀.₅, with OOD generalization emerging as the data volume grows.
~38,100h全开源预训练语料（含 H2R 合成）Fully open-source pre-training corpus (incl. H2R synthesis)
15种双臂机器人形态（人机合成覆盖）dual-arm robot embodiments (covered by human-to-robot synthesis)
+20%RoboChallenge Table30-v1 Generalist Track 相对提升（vs. DM0_generalist）Relative gain on the RoboChallenge Table30-v1 Generalist Track (vs. DM0_generalist)
为什么标准基准不够用？Why are standard benchmarks not enough?
在标准 in-distribution 基准（LIBERO、RoboTwin）上，从零训练的模型（StarVLA、Ours-scratch）可媲美甚至超过大规模预训练模型（π₀、π₀.₅）。这是因为训练与评测共享同一分布，高准确率可通过记忆视觉-行为模式实现，而非真实泛化能力。
相比之下，在 OOD 基准（LIBERO-Plus、RoboTwin-Clean2Rand）上，差距显著拉开：π₀.₅ 大幅优于从零训练的模型，随扰动难度上升差距进一步加大。因此 Qwen-RobotManip 以 OOD 评测为首要指标。On standard in-distribution benchmarks (LIBERO, RoboTwin), models trained from scratch (StarVLA, Ours-scratch) match or even exceed large-scale pre-trained models (π₀, π₀.₅). This is because training and evaluation share the same distribution, so high accuracy can be reached by memorizing visual-behavioral patterns rather than by genuine generalization.
By contrast, on OOD benchmarks (LIBERO-Plus, RoboTwin-Clean2Rand) the gap opens up sharply: π₀.₅ substantially outperforms models trained from scratch, and the gap widens further as perturbation difficulty rises. Qwen-RobotManip therefore takes OOD evaluation as its primary metric.
02方法Method
Qwen-RobotManip 围绕「对齐优先」原则，在数据、模型、训练三个层面协同设计：统一 80 维规范状态-动作表示、相机坐标系 delta 姿态运动对齐、结构化体态提示与上下文内策略自适应，配合 Human-to-Robot 合成流水线实现大规模多源训练。Qwen-RobotManip is built around an "alignment-first" principle, with coordinated design at the data, model and training levels: a unified 80-dimensional canonical state-action representation, camera-frame delta-pose motion alignment, structured embodiment prompts with in-context policy adaptation, and a Human-to-Robot synthesis pipeline that enables large-scale multi-source training.
图3 Qwen-RobotManip 模型架构概览。
模型耦合 Qwen-VL 主干（Qwen3.5-4B）与 flow-matching Diffusion Transformer (DiT) 动作头。主干联合编码多视角视觉 token、结构化体态提示（Embodied Chain-of-Thought）、历史上下文 token，最后层隐藏态通过交替 cross-attention 注入 DiT。状态与动作共享统一 80 维规范表示，EEF 动作以相机坐标系 delta 姿态表达，条件于相机参数与 end-effector 类型嵌入。VLM 协同训练与 VLA 训练使用独立 batch。Figure 3 Overview of the Qwen-RobotManip model architecture.
The model couples a Qwen-VL backbone (Qwen3.5-4B) with a flow-matching Diffusion Transformer (DiT) action head. The backbone jointly encodes multi-view visual tokens, structured embodiment prompts (Embodied Chain-of-Thought) and historical context tokens; last-layer hidden states are injected into the DiT through interleaved cross-attention. States and actions share a unified 80-dimensional canonical representation, EEF actions are expressed as camera-frame delta poses, conditioned on camera parameters and end-effector type embeddings. VLM co-training and VLA training use separate batches.
表示对齐：统一状态-动作空间Representation alignment: a unified state-action space
引入 80 维规范向量表示：两个 29 维单臂 block（各含 Joint positions 7d、EEF pose 9d、Gripper state 1d、Dexterous hand joints 12d）+ 22 维预留维度。不同形态（单臂 Franka、双臂 ALOHA、人形机器人）各自填充对应子集，未激活维度通过 per-dimension binary mask 排除在训练损失之外，避免 "幽灵" 监督干扰梯度。Introduces an 80-dimensional canonical vector representation: two 29-dimensional single-arm blocks (each holding Joint positions 7d, EEF pose 9d, Gripper state 1d, Dexterous hand joints 12d) plus 22 reserved dimensions. Different embodiments (single-arm Franka, dual-arm ALOHA, humanoid robots) fill in their own subsets, and inactive dimensions are excluded from the training loss by a per-dimension binary mask, avoiding "ghost" supervision that would perturb the gradients.
不同数据集的 end-effector 姿态在不同坐标系下记录，导致即使是相同物理运动，数值上也可能截然不同。
通过将 EEF delta 动作表示为相机坐标系 delta pose（camera-frame delta pose），视觉上相似的运动在动作空间中数值也相近，直接对齐视觉观察空间与动作空间，促进跨形态迁移。
同时通过 Camera-aware Positional Encoding (CaPE) 将相机外参注入 DiT cross-attention，使动作头能够推理相机几何关系。End-effector poses are recorded in different coordinate frames across datasets, so even identical physical motions can take drastically different numerical values.
By expressing EEF delta actions as a camera-frame delta pose, visually similar motions also take similar values in action space, which directly aligns the visual observation space with the action space and promotes cross-embodiment transfer.
Camera extrinsics are additionally injected into the DiT cross-attention through Camera-aware Positional Encoding (CaPE), letting the action head reason about camera geometry.
每步推断时提供结构化文本提示，字段包括：embodiment（机器人平台标识）、instruction（任务描述）、speed（episode 长度 bin，500 步为单位）、fps（时间采样率）、camera view direction（arm side / opposite side）。为提高对缺失信息的鲁棒性，训练中以 15% 概率随机丢弃 embodiment、speed、fps 字段。A structured text prompt is supplied at every inference step, with the fields embodiment (robot platform identifier), instruction (task description), speed (episode-length bin, in units of 500 steps), fps (temporal sampling rate) and camera view direction (arm side / opposite side). To improve robustness against missing information, the embodiment, speed and fps fields are dropped at random with 15% probability during training.
上下文内策略自适应（In-Context）In-context policy adaptation
受大语言模型 in-context learning 启发，Qwen-RobotManip 配备上下文策略自适应机制：将同一 episode 内近期执行历史（observation-action chunk 对）作为结构化上下文 token 序列注入策略，无需更新参数即可在部署时实现行为自适应。采用随机上下文采样（Stochastic context sampling）防止模型退化为简单的"复制近期动作"捷径。Inspired by in-context learning in large language models, Qwen-RobotManip is equipped with an in-context policy adaptation mechanism: the recent execution history within the same episode (observation-action chunk pairs) is injected into the policy as a structured sequence of context tokens, so that behavioral adaptation is achieved at deployment time without updating parameters. Stochastic context sampling is used to keep the model from degenerating into the trivial "copy the recent actions" shortcut.
Human-to-Robot 数据合成流水线Human-to-Robot data synthesis pipeline
自我中心人手演示与机器人数据在形态和视觉域之间存在显著差距。合成流水线分两阶段：There is a substantial gap between egocentric human-hand demonstrations and robot data, both in embodiment and in visual domain. The synthesis pipeline runs in two stages:
Action Alignment：将人手关键点（MANO 表示，21 个关节）映射到并联夹爪机器人动作空间，定义虚拟指尖 k_vf = 0.7·k_index + 0.3·k_middle，EEF 位置取拇指与虚拟指尖中点，夹爪宽度为欧氏距离。应用 Savitzky-Golay 滤波平滑轨迹，高斯加权 SLERP 平滑方向。Action Alignment: human-hand keypoints (MANO representation, 21 joints) are mapped into the action space of a parallel-jaw gripper robot; a virtual fingertip is defined as k_vf = 0.7·k_index + 0.3·k_middle, the EEF position is the midpoint between thumb and virtual fingertip, and the gripper width is their Euclidean distance. A Savitzky-Golay filter smooths the trajectories and Gaussian-weighted SLERP smooths the orientations.
Visual Alignment：SAM3 分割人手区域，ProPainter 光流引导修复背景，MuJoCo IK 求解机器人底座位姿，Depth Anything v3 估计场景深度后合成机器人图像。Visual Alignment: SAM3 segments the human-hand region, ProPainter inpaints the background under optical-flow guidance, MuJoCo IK solves for the robot base pose, and Depth Anything v3 estimates scene depth before the robot image is composited.
~1,933 小时自我中心数据渲染为 15 种双臂机器人形态（Panda、UR5e、ARX-L5、xArm7、IIWA、Kinova Gen3 等），共约 24,808 小时合成演示。~1,933 hours of egocentric data are rendered into 15 dual-arm robot embodiments (Panda, UR5e, ARX-L5, xArm7, IIWA, Kinova Gen3 and others), yielding about 24,808 hours of synthetic demonstrations.
以 9:1 的机器人数据与 VL 数据比例同步训练 VLA 流和 VLM 流，防止动作预测压力导致 VLM 主干感知与推理能力退化。VL 数据包括通用视觉理解、空间感知与推理、OCR、多模态专业知识、指令跟随，以及专门合成的体态中心 VL 数据（Embodied Chain-of-Thought、自我中心视频理解、2D 轨迹预测）。The VLA and VLM streams are trained in parallel at a 9:1 ratio of robot data to VL data, preventing the pressure of action prediction from degrading the perception and reasoning ability of the VLM backbone. The VL data covers general visual understanding, spatial perception and reasoning, OCR, multimodal domain knowledge and instruction following, together with purpose-built embodiment-centric VL data (Embodied Chain-of-Thought, egocentric video understanding, 2D trajectory prediction).
03实验Experiments
在仿真 OOD 基准（LIBERO-Plus、RoboTwin-Clean2Rand、RoboCasa365、EBench、RoboTwin-IF、RoboTwin-XE）与真实机器人平台（AgileX CobotMagic ALOHA、ARX ALOHA、UR5、Franka）上进行系统评测，以 π₀（Black et al., 2024）和 π₀.₅（Black et al., 2025）为主要基线。Systematic evaluation is carried out on simulated OOD benchmarks (LIBERO-Plus, RoboTwin-Clean2Rand, RoboCasa365, EBench, RoboTwin-IF, RoboTwin-XE) and on real robot platforms (AgileX CobotMagic ALOHA, ARX ALOHA, UR5, Franka), with π₀ (Black et al., 2024) and π₀.₅ (Black et al., 2025) as the main baselines.
图7 OOD 泛化结果汇总（论文 Figure 7）。
(a) 任务与场景泛化：LIBERO-Plus、EBench、RoboCasa365-Unseen、LIBERO-Plus 各设定下 Qwen-RobotManip（深紫）大幅领先 π₀.₅（浅紫），且差距随评测难度上升而扩大。
(b) 指令跟随（RoboTwin-IF）：Pick-Diverse（+35pp）、Place-Relative（+37pp）、Operate-Mic-Drawer（+27pp）、Operate-Tabletop（+27pp）均显著领先，平均 72.2% vs. π₀.₅ 49.6%。
(c) 零样本跨形态迁移（RoboTwin-XE，camera-frame EEF）：ARX-X5 42.9%、UR5-WSG 22.8%、Franka 5.9%，总体 23.9%，约为 π₀.₅（7.5%）的 3.2 倍。Figure 7 Summary of OOD generalization results (paper Figure 7).
(a) Task and scene generalization: across the LIBERO-Plus, EBench, RoboCasa365-Unseen and LIBERO-Plus settings, Qwen-RobotManip (dark purple) leads π₀.₅ (light purple) by a wide margin, and the gap grows as evaluation difficulty rises.
(b) Instruction following (RoboTwin-IF): Pick-Diverse (+35pp), Place-Relative (+37pp), Operate-Mic-Drawer (+27pp) and Operate-Tabletop (+27pp) all lead clearly, averaging 72.2% vs. 49.6% for π₀.₅.
(c) Zero-shot cross-embodiment transfer (RoboTwin-XE, camera-frame EEF): ARX-X5 42.9%, UR5-WSG 22.8%, Franka 5.9%, 23.9% overall—about 3.2 times that of π₀.₅ (7.5%).
基准（OOD）Benchmark (OOD)
π₀.₅ (SOTA)
Qwen-RobotManip
Qwen-RobotManip-Context
LIBERO-Plus (Total)
84.4
89.0
91.4
RoboTwin-C2R (Hard)
47.9
69.4
69.4
RoboCasa365 (Total)
16.9
35.9
33.8
EBench (Overall SR)
27.1
45.6
43.6
RoboTwin-IF (Avg.)
49.6
72.2
72.0
RoboTwin-XE (Avg., eef)
7.5
23.9
—
真实机器人评测Real-robot evaluation
图17 RoboChallenge Table30-v1 六项高难度任务（论文 Figure 17）。
包括：插网线（30% vs. 先前 SOTA 10%）、倒薯条入盘（30% vs. 0%）、制作素食三明治（10% vs. 0%——唯一非零成功率模型）、排列纸杯（70% vs. 10%）、分拣电子产品（50% vs. 0%）、折叠洗碗布（30% vs. 10%）。先前 SOTA Generalist 平均仅 5%，Qwen-RobotManip 达 36.7%。Figure 17 Six high-difficulty tasks of RoboChallenge Table30-v1 (paper Figure 17).
They include: inserting a network cable (30% vs. 10% for the prior SOTA), pouring fries onto a plate (30% vs. 0%), making a vegetarian sandwich (10% vs. 0%—the only model with a non-zero success rate), arranging paper cups (70% vs. 10%), sorting electronic products (50% vs. 0%) and folding a dishcloth (30% vs. 10%). The prior SOTA Generalist averages only 5%, while Qwen-RobotManip reaches 36.7%.
在 CobotMagic ALOHA 平台上，ID 基准平均成功率 88.6%（π₀.₅ 42.9%，StarVLA 20.0%），OOD 基准 87.5%（π₀.₅ 37.5%，StarVLA 0.0%）。在 ARX ALOHA 上，少样本自适应（130 次演示）四项任务 Qwen-RobotManip 胜出；跨形态技能迁移（4 项新任务，零 ARX 演示）达 55.0%，而消融基线（w/o UnifiedSpace 7.5%、w/o UnifiedEEF 12.5%）几乎失败，验证统一对齐的关键作用。On the CobotMagic ALOHA platform the average ID benchmark success rate is 88.6% (π₀.₅ 42.9%, StarVLA 20.0%) and the OOD benchmark reaches 87.5% (π₀.₅ 37.5%, StarVLA 0.0%). On ARX ALOHA, Qwen-RobotManip wins on all four few-shot adaptation tasks (130 demonstrations); cross-embodiment skill transfer (4 new tasks, zero ARX demonstrations) attains 55.0%, while the ablation baselines (w/o UnifiedSpace 7.5%, w/o UnifiedEEF 12.5%) almost entirely fail, confirming the critical role of unified alignment.
RoboChallenge Table30-v1 Generalist Track
提交 Qwen-RobotManip 至 Table30-v1 Generalist Track（30 项操作任务，跨 4 个实体形态），成功率 45%，过程分 59.83，超越 DM0_generalist（37% / 48.43 分）8 个百分点，排名第一，相对提升约 20%。在双臂协调任务（8 项）上平均 40.0%，在拾放任务（12 项）上平均 63.3%，均大幅领先所有基线。Qwen-RobotManip was submitted to the Table30-v1 Generalist Track (30 manipulation tasks across 4 physical embodiments), reaching a 45% success rate and a process score of 59.83, ahead of DM0_generalist (37% / 48.43 points) by 8 percentage points and ranking first, a relative gain of about 20%. It averages 40.0% on dual-arm coordination tasks (8 tasks) and 63.3% on pick-and-place tasks (12 tasks), leading all baselines by a wide margin.
消融实验Ablation studies
数据规模化曲线（Figure 18）显示：具有统一表示的变体（Ours、Ours w/o UnifiedEEF）在 1%–100% 训练数据区间内验证 MSE 近似对数线性下降，呈清晰数据规模化律；而 Ours w/o UnifiedSpace 曲线不稳定，EEF 动作预测 MSE 显著更高。验证了「对齐是规模化的前提」这一核心命题。The data-scaling curves (Figure 18) show that the variants with a unified representation (Ours, Ours w/o UnifiedEEF) exhibit an approximately log-linear decrease in validation MSE over the 1%–100% training-data range, a clear data-scaling law, whereas the Ours w/o UnifiedSpace curve is unstable and its EEF action-prediction MSE is markedly higher. This validates the core claim that "alignment is the prerequisite for scaling".
上下文自适应消融（Table 15）：无上下文基线（Structure Prompt 配置）Hard 平均 65.9%；添加上下文（10 denoising steps）提升至 70.9%（+5.0 点）；20 steps 无额外收益（71.0%）。随机上下文采样对于防止"复制近期动作"退化至关重要。In-context adaptation ablation (Table 15): the context-free baseline (Structure Prompt configuration) averages 65.9% on Hard; adding context (10 denoising steps) raises this to 70.9% (+5.0 points); 20 steps brings no further gain (71.0%). Stochastic context sampling is essential for preventing degeneration into "copying the recent actions".
04局限性Limitations
说明：以下局限性部分为作者在技术报告中明确陈述，部分为从设计中推断（标注"设计推断"）。Note: some of the limitations below are explicitly stated by the authors in the technical report, others are inferred from the design (marked "design inference").
cold-start 上下文问题（作者明确陈述）Cold-start context problem (explicitly stated by the authors)
「We do note one practical limitation observed in real-robot deployment: at the start of an episode, the context consists entirely of zero-padded placeholders, and the model, having learned to condition on quiescent history, tends to hesitate before initiating motion.」为此同时发布有/无上下文两种变体供用户按需选择。"We do note one practical limitation observed in real-robot deployment: at the start of an episode, the context consists entirely of zero-padded placeholders, and the model, having learned to condition on quiescent history, tends to hesitate before initiating motion." Both a context and a context-free variant are therefore released, so users can pick whichever they need.
精细操作任务仍有明显差距（作者明确陈述）A clear gap remains on fine-grained manipulation tasks (explicitly stated by the authors)
在 CobotMagic ALOHA ID 基准中，yellow-disc-insertion（精确插盘）成功率仅 2/5，作者指出「highlighting the difficulty of precise insertion on real hardware」。高精度接触丰富型操作仍是挑战。On the CobotMagic ALOHA ID benchmark, yellow-disc-insertion (precise disc insertion) succeeds only 2/5 of the time, and the authors note "highlighting the difficulty of precise insertion on real hardware". High-precision, contact-rich manipulation remains a challenge.
数据规模依赖合成与人手视频，领域偏移风险（设计推断）Data scale depends on synthesis and human-hand video, with domain-shift risk (design inference)
~24,808 小时合成数据来自人手视频经 Human-to-Robot 流水线渲染，视觉质量和动作分布与真实机器人数据仍存在域差距。尽管合成流水线已引入深度合成和 IK 求解，但极端光照、复杂纹理等场景的泛化能力尚未完整评估。The ~24,808 hours of synthetic data are rendered from human-hand video through the Human-to-Robot pipeline, and a domain gap in visual quality and action distribution relative to real robot data still remains. Although the synthesis pipeline already brings in depth synthesis and IK solving, generalization to scenarios such as extreme lighting and complex textures has not yet been fully evaluated.
评测覆盖局限：移动操作、灵巧手等形态偏少（设计推断）Limited evaluation coverage: few mobile-manipulation and dexterous-hand embodiments (design inference)
当前 OOD 评测主要集中在固定基座的桌面操作（单臂/双臂）。EBench 中移动操作任务上 Qwen-RobotManip 成功率（≈35%）低于固定桌面任务（≈50%），灵巧手形态数据量相对有限，泛化能力尚待验证。Current OOD evaluation concentrates on fixed-base tabletop manipulation (single-arm / dual-arm). On the EBench mobile-manipulation tasks the success rate of Qwen-RobotManip (≈35%) is lower than on fixed tabletop tasks (≈50%); data for dexterous-hand embodiments is relatively limited, and generalization there remains to be verified.
