Qwen-VLA：跨任务、环境与机器人形态的统一视觉-语言-行动建模Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments
Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments
Qwen Team（通义团队）· 阿里巴巴 · arXiv:2605.30280 · 2026年6月Qwen Team (Tongyi Lab) · Alibaba · arXiv:2605.30280 · June 2026
Qwen-VLA 是首个将机器人操作、室内导航与轨迹预测统一在单一 Vision-Language-Action（VLA）模型中的具身基础模型。它在 Qwen3.5-4B 多模态骨干之上附加一个 DiT 流匹配（flow matching）动作解码器，通过形态感知提示条件（embodiment-aware prompt conditioning）和大规模联合预训练，实现单模型跨多平台部署——作为通才策略，同时在操作、导航、零样本泛化等多个基准上超越专家模型。Qwen-VLA is the first embodied foundation model to unify robot manipulation, indoor navigation, and trajectory prediction within a single Vision-Language-Action (VLA) model. On top of the Qwen3.5-4B multimodal backbone it attaches a DiT flow-matching action decoder, and through embodiment-aware prompt conditioning and large-scale joint pretraining it achieves single-model deployment across multiple platforms — as a generalist policy, it simultaneously surpasses specialist models on manipulation, navigation, zero-shot generalization, and other benchmarks.
具身智能的研究长期依赖专用模型：操作模型、导航模型各自独立，导致能力碎片化、跨场景泛化能力受限。这些模型表面上异构——动作空间、控制频率、预测视野均不相同——却共享同一底层计算结构：从视觉观测和语言指令出发，预测物理上合理的未来动作或轨迹。Qwen-VLA 正是基于这一观察，探索能否将操作、导航、轨迹预测等异构具身决策问题统一到单一 VLA 模型中。Research on embodied intelligence has long relied on specialized models: manipulation models and navigation models are built independently, which fragments capabilities and limits cross-scenario generalization. These models are heterogeneous on the surface — action spaces, control frequencies, and prediction horizons all differ — yet they share the same underlying computational structure: starting from visual observations and language instructions, predict physically plausible future actions or trajectories. Building on this observation, Qwen-VLA explores whether heterogeneous embodied decision-making problems such as manipulation, navigation, and trajectory prediction can be unified within a single VLA model.
"We investigate whether these heterogeneous embodied decision-making problems can be unified within a single vision-language-action model."
图1：Qwen-VLA 总览。模型以 Qwen3.5 视觉-语言骨干为基础，附加 DiT 流匹配动作解码器，在混合操作（VLA）、导航（VLN）和视觉-语言理解数据上联合训练，统一生成机器人动作与文本响应。Figure 1: Overview of Qwen-VLA. The model builds on the Qwen3.5 vision-language backbone and attaches a DiT flow-matching action decoder, trained jointly on mixed manipulation (VLA), navigation (VLN), and vision-language understanding data to generate robot actions and text responses in a unified way.
76.9%ALOHA 真机 OOD 平均成功率ALOHA real-robot OOD average success rate
上述数字均来自论文表格，Qwen-VLA-Instruct 作为通才策略（generalist policy）一次性训练，无需针对各基准单独微调，在 LIBERO、RoboTwin-Easy/Hard（86.1%/87.2%）、RxR 导航（59.6% SR）等多个基准上与甚至超越专用专家模型。All the numbers above are taken from tables in the paper. Trained once as a generalist policy, with no per-benchmark fine-tuning, Qwen-VLA-Instruct matches or even surpasses dedicated specialist models on benchmarks such as LIBERO, RoboTwin-Easy/Hard (86.1%/87.2%), and RxR navigation (59.6% SR).
02方法Method
Qwen-VLA 的核心由三部分构成：(1) Qwen3.5-4B 多模态视觉-语言骨干；(2) 单流 DiT 流匹配动作解码器；(3) 统一动作与轨迹表示。通过形态感知提示条件（embodiment-aware prompt conditioning）在无需改变模型架构的前提下支持多机器人平台，并采用四阶段渐进训练流程（T2A → CPT → SFT → RL）将动作先验学习、视觉对齐、任务特化和成功率优化分离成独立阶段。Qwen-VLA has three core parts: (1) a Qwen3.5-4B multimodal vision-language backbone; (2) a single-stream DiT flow-matching action decoder; (3) a unified action and trajectory representation. Embodiment-aware prompt conditioning supports multiple robot platforms without changing the model architecture, and a four-stage progressive training pipeline (T2A → CPT → SFT → RL) separates action-prior learning, visual alignment, task specialization, and success-rate optimization into independent stages.
图2：Qwen-VLA 训练流程。Stage I（T2A）仅用文本-动作数据训练 DiT 解码器，建立语言索引的动作先验；Stage II（CPT）解冻两个模块，在异构数据上进行视觉对齐；Stage III（SFT）分为多任务和真机两条并行轨迹；Stage IV（RL）在仿真环境（SimplerEnv）中用稀疏二值奖励优化闭环成功率，得到最终模型 Qwen-VLA-Instruct。Figure 2: The Qwen-VLA training pipeline. Stage I (T2A) trains only the DiT decoder on text-action data, establishing a language-indexed action prior; Stage II (CPT) unfreezes both modules and performs visual alignment on heterogeneous data; Stage III (SFT) splits into two parallel tracks, multi-task and real-robot; Stage IV (RL) optimizes closed-loop success rate with a sparse binary reward in a simulation environment (SimplerEnv), yielding the final model Qwen-VLA-Instruct.
视觉-语言骨干：Qwen3.5Vision-Language Backbone: Qwen3.5
骨干采用 Qwen3.5（Team, 2026），原生多模态模型，通过 ViT + 空间合并将视觉 token 直接插入文本 token 流，采用混合注意力设计（门控线性注意力 + 分组查询 softmax 注意力），在保留全精度全局推理的同时高效编码长多模态序列。The backbone is Qwen3.5 (Team, 2026), a natively multimodal model that inserts visual tokens directly into the text token stream via ViT plus spatial merging, and adopts a hybrid attention design (gated linear attention plus grouped-query softmax attention) to encode long multimodal sequences efficiently while retaining full-precision global reasoning.
DiT 流匹配动作解码器DiT Flow-Matching Action Decoder
动作专家（action expert）采用单流 DiT 风格的流匹配策略。它将 VLM 隐状态与带噪动作块拼接成一个序列，经过带 AdaLN 时间步条件的联合 self-attention 处理，参数约 1.15B（16 个 DiT 块，每块 70.8M）。训练时采用流匹配目标（flow-matching objective），推理时通过少量 Euler 积分步从 τ=1 到 τ=0 生成动作块，实现低延迟实时控制。The action expert adopts a single-stream DiT-style flow-matching policy. It concatenates the VLM hidden states and the noisy action chunk into a single sequence, processed by joint self-attention with AdaLN timestep conditioning, for roughly 1.15B parameters (16 DiT blocks, 70.8M each). Training uses a flow-matching objective; at inference a small number of Euler integration steps run from τ=1 to τ=0 to generate an action chunk, enabling low-latency real-time control.
每个训练样本前置一个描述当前平台、手臂配置、控制约定的文本提示：Every training sample is prefixed with a text prompt describing the current platform, arm configuration, and control conventions:
"The robot is {robot_tag} with {single arm / dual arms}[, waist][, and mobile base]. The control frequency is {FPS} Hz. Please predict the next {chunk_size} control actions to execute the following task: {ori_instruction}."
该提示是模型了解平台专属控制语义的唯一接口，无需额外的形态专用输出头。预训练语料覆盖 WidowX、Google Robot、Franka Panda、Fourier GR-1、Mobile ALOHA、AgiBot A2-D 等 11 种代表性机器人平台。This prompt is the model's only interface for platform-specific control semantics; no extra embodiment-specific output head is needed. The pretraining corpus covers 11 representative robot platforms, including WidowX, Google Robot, Franka Panda, Fourier GR-1, Mobile ALOHA, and AgiBot A2-D.
统一动作与轨迹表示Unified Action and Trajectory Representation
所有任务输出均表示为 Y ∈ RH×K（H 为预测视野，K 为共享通道维度）。操作任务（∆EEF、关节角度、夹手状态）和导航任务（∆x, ∆y, ∆θ 航点）均占据 Y 的前 c ≤ K 个通道，其余通道零填充，由二值掩码 M ∈ {0,1}H×K 屏蔽填充项对梯度的影响。这使单一 DiT 参数集可处理所有控制模式。Every task output is represented as Y ∈ RH×K (H is the prediction horizon, K the shared channel dimension). Manipulation tasks (∆EEF, joint angles, gripper state) and navigation tasks (∆x, ∆y, ∆θ waypoints) both occupy the first c ≤ K channels of Y, with the remaining channels zero-padded and a binary mask M ∈ {0,1}H×K blocking padded entries from affecting gradients. This lets a single set of DiT parameters handle every control mode.
四阶段渐进训练Four-Stage Progressive Training
Stage I — T2A（Text-to-Action 预训练）：冻结 VLM，仅训练 DiT，刻意屏蔽图像，迫使解码器从紧凑的语言描述中重建高维动作分布，建立语言索引的动作先验。最优配置：∼20% 合成 + 80% 真实数据，全序列预测，Sigmoid-Normal 时间步分布，2,000 步，较无 T2A 基线提升 +10.2 pp（71.1% vs. 60.9%）。Stage I — T2A (Text-to-Action pretraining): Freeze the VLM and train only the DiT, deliberately masking out images so the decoder must reconstruct the high-dimensional action distribution from compact language descriptions, establishing a language-indexed action prior. Best configuration: ∼20% synthetic + 80% real data, full-sequence prediction, a Sigmoid-Normal timestep distribution, 2,000 steps — an improvement of +10.2 pp over the no-T2A baseline (71.1% vs. 60.9%).
Stage II — CPT（持续预训练）：解冻两个模块，在包括机器人操作轨迹（74.2%）、人类第一视角演示（6.0%）、导航数据（7.5%）、合成仿真数据（3.7%）和辅助视觉-语言数据（8.5%）的混合语料上训练，使骨干适应具身感知。Stage II — CPT (continual pretraining): Unfreeze both modules and train on a mixed corpus of robot manipulation trajectories (74.2%), human egocentric demonstrations (6.0%), navigation data (7.5%), synthetic simulation data (3.7%), and auxiliary vision-language data (8.5%), adapting the backbone to embodied perception.
Stage III — SFT（监督微调）：从 CPT 检查点出发，分两条并行轨迹：多任务 SFT（联合 VQA、空间感知、操作、导航）和真机轨迹（ALOHA 遥操作数据）。损失权重：视觉-语言 0.1，动作 1.0。Stage III — SFT (supervised fine-tuning): Starting from the CPT checkpoint, split into two parallel tracks: multi-task SFT (jointly covering VQA, spatial perception, manipulation, and navigation) and real-robot trajectories (ALOHA teleoperation data). Loss weights: 0.1 for vision-language, 1.0 for action.
Stage IV — RL（强化学习）：从多任务 SFT 检查点出发，使用 PPO + GAE 在 SimplerEnv 中以稀疏二值成功奖励优化闭环策略，N=128 并行环境，每次迭代 8,192 个动作块过渡，得到最终 Qwen-VLA-Instruct。Stage IV — RL (reinforcement learning): Starting from the multi-task SFT checkpoint, PPO + GAE optimizes the closed-loop policy in SimplerEnv with a sparse binary success reward, using N=128 parallel environments and 8,192 action-chunk transitions per iteration, yielding the final Qwen-VLA-Instruct.
预训练数据规模Pretraining Data Scale
机器人操作轨迹（含 10,000+ 小时公开数据集 + 1,000+ 小时自有数据）、8M+ 合成仿真轨迹（IsaacLab + cuRobo）、EgoDex（829 小时）、EgoVerse（1,300+ 小时）、多种室内导航数据集，以及约 48,000 个细粒度具身动作标注 video-caption 对。Robot manipulation trajectories (10,000+ hours of public datasets plus 1,000+ hours of in-house data), 8M+ synthetic simulation trajectories (IsaacLab + cuRobo), EgoDex (829 hours), EgoVerse (1,300+ hours), a range of indoor navigation datasets, and roughly 48,000 video-caption pairs with fine-grained embodied action annotations.
03实验Experiments
实验覆盖机器人操作（仿真+真机）、室内视觉-语言导航、静态与动态操作 OOD 泛化共四大场景，比较两个模型变体：Qwen-VLA-Base（大规模预训练后）和 Qwen-VLA-Instruct（SFT+RL 后）。The experiments span four major settings — robot manipulation (simulation and real robots), indoor vision-language navigation, and static and dynamic manipulation OOD generalization — comparing two model variants: Qwen-VLA-Base (after large-scale pretraining) and Qwen-VLA-Instruct (after SFT+RL).
Qwen-VLA-Instruct 作为通才模型，在 LIBERO 达到 97.9%，与最优专家模型持平；在 Simpler-WidowX（73.7%）、RoboTwin-Easy（86.1%）、RoboTwin-Hard（87.2%）上超越所有专家模型。RoboCasa-GR1 达到 56.7%，超越 π0.5（37.0%）、GR00T N1.6（49.9%）和 Being-H0.5（53.3%）。As a generalist model, Qwen-VLA-Instruct reaches 97.9% on LIBERO, on par with the best specialist model, and surpasses every specialist on Simpler-WidowX (73.7%), RoboTwin-Easy (86.1%), and RoboTwin-Hard (87.2%). On RoboCasa-GR1 it reaches 56.7%, beating π0.5 (37.0%), GR00T N1.6 (49.9%), and Being-H0.5 (53.3%).
图4：真机评测在 ALOHA 双臂平台上的 6 类任务（域内）及 5 类 OOD 泛化设置（颜色/实例/位置/背景/指令泛化）的示意图。左列展示域内任务（拾取放置、清台面、碗叠放、细粒度操作等），右列展示对应 OOD 变体。Figure 4: Real-robot evaluation on the ALOHA bimanual platform — 6 task categories (in-domain) and 5 OOD generalization settings (color / instance / position / background / instruction generalization). The left column shows in-domain tasks (pick-and-place, table clearing, bowl stacking, fine-grained manipulation, and more), the right column the corresponding OOD variants.
模型Model
域内平均（%）In-domain avg. (%)
OOD 平均（%）OOD avg. (%)
GR00T N1.6
28.6
25.4
π0.5
71.6
41.5
Qwen-VLA-alohaw/o pretrain
48.5
36.2
Qwen-VLA-alohaw/ pretrain
83.6
76.9
从 Qwen-VLA-Base 微调的版本（有预训练）将域内平均成功率从 48.5% 提升至 83.6%，OOD 平均达 76.9%，较 π0.5 高 35.4 个百分点，较无预训练版本高 40.7 个百分点。The version fine-tuned from Qwen-VLA-Base (with pretraining) raises the in-domain average success rate from 48.5% to 83.6% and reaches 76.9% OOD average — 35.4 percentage points above π0.5 and 40.7 percentage points above the version without pretraining.
Qwen-VLA-Instruct 以 26.6% SR、39.5 MS 成为 DOMINO 动态操作基准最优，超越专门在动态操作数据上微调的 PUMA（SR 17.2%，MS 35.0）9.4 pp，且完全未使用动态操作训练数据。With 26.6% SR and 39.5 MS, Qwen-VLA-Instruct is the best model on the DOMINO dynamic manipulation benchmark, exceeding PUMA — fine-tuned specifically on dynamic manipulation data (SR 17.2%, MS 35.0) — by 9.4 pp, while using no dynamic manipulation training data at all.
消融实验Ablations
图6：T2A 预训练消融。(a) 数据配比与预测模式：∼20% 合成 + 80% 真实数据 + 全序列预测最优（71.1%），无 T2A 基线为 60.9%（+10.2 pp）；图像输入在 T2A 阶段有害（−2.9 pp）。(b) 时间步分布：T2A 用 Sigmoid-Normal、SFT 用 Beta 组合最优（71.1%）。(c) T2A 训练步数：2,000 步最优，40,000 步出现过拟合。Figure 6: T2A pretraining ablations. (a) Data mixture and prediction mode: ∼20% synthetic + 80% real data with full-sequence prediction is best (71.1%), against 60.9% for the no-T2A baseline (+10.2 pp); image input is harmful during the T2A stage (−2.9 pp). (b) Timestep distribution: Sigmoid-Normal for T2A combined with Beta for SFT is best (71.1%). (c) T2A training steps: 2,000 steps is best, while 40,000 steps overfits.
视觉-语言联合训练在需要细粒度识别和组合指令解析的基准上带来显著提升：RoboCasa-GR1 +4.9 pp（51.1% → 56.0%），RoboTwin-2.0 +4.6 pp（81.8% → 86.4%）。零填充（Zero-Padding）投影设计与 Multi-MLP、Concatenation 性能相当（差距 <1.2 pp），但参数量最少，成为默认选择。RL 后训练在 SimplerEnv 目标环境提升 +2.9 pp，且在未参与 RL 训练的基准上性能保持或温和提升，无灾难性遗忘。Joint vision-language training brings clear gains on benchmarks that require fine-grained recognition and compositional instruction parsing: RoboCasa-GR1 +4.9 pp (51.1% → 56.0%) and RoboTwin-2.0 +4.6 pp (81.8% → 86.4%). The Zero-Padding projection design performs on par with Multi-MLP and Concatenation (gap <1.2 pp) while using the fewest parameters, making it the default choice. RL post-training improves the SimplerEnv target environment by +2.9 pp, and performance on benchmarks not involved in RL training is preserved or mildly improved, with no catastrophic forgetting.
04局限性Limitations
Note：以下三点均为论文第 7 节（Limitations and Future Work）作者明确陈述（stated），非推断。Note: All three points below are explicitly stated by the authors in Section 7 (Limitations and Future Work) of the paper; none are inferred.
具身动作数据规模与多样性不足Embodied action data is limited in scale and diversity
"Embodied action data remains far smaller and less diverse than vision-language pretraining data, limiting robustness to long-tail objects, environments, embodiments, and contact-rich interactions."——长尾物体、接触丰富的交互（如布料折叠、精细插接）仍是薄弱环节。— Long-tail objects and contact-rich interactions (such as cloth folding and precise insertion) remain weak spots.
多任务联合训练的优化权衡Optimization trade-offs in multi-task joint training
"Joint training across vision-language understanding, navigation, and action generation introduces optimization trade-offs. While action-oriented training improves policy learning, it can modestly regress some pure vision-language and navigation evaluations, suggesting the need for better objective balancing, data curricula, and modular specialization."——动作学习与语言理解之间存在梯度竞争，需要更好的目标平衡和数据课程。— Gradient competition exists between action learning and language understanding, calling for better objective balancing and data curricula.
评估仍以短视野、基准驱动为主Evaluation remains largely short-horizon and benchmark-driven
"Current evaluations are still largely short-horizon and benchmark-driven, leaving long-duration, failure-prone real-world deployment as an open challenge."——长时程规划、失败恢复、开放世界真机部署仍是未解决的挑战。未来方向包括情节记忆、世界模型预测、力觉/触觉等更丰富的物理反馈。— Long-horizon planning, failure recovery, and open-world real-robot deployment remain unsolved challenges. Future directions include episodic memory, world-model prediction, and richer physical feedback such as force and tactile sensing.
