面向真实世界部署的动作推理模型Action Reasoning Models for Real-World Deployment
Haoquan Fang, Jiafei Duan, Donovan Clay, Sam Wang, Shuo Liu, Weikai Huang, Xiang Fan 等 · Allen Institute for AI & University of WashingtonHaoquan Fang, Jiafei Duan, Donovan Clay, Sam Wang, Shuo Liu, Weikai Huang, Xiang Fan et al. · Allen Institute for AI & University of Washington
MolmoAct2 是一个完全开源的 Vision-Language-Action (VLA) 模型，围绕五个维度全面超越前作 MolmoAct：更强的具身推理 VLM 骨干 Molmo2-ER、三个新的机器人数据集（含迄今最大的开源双臂数据集 720 小时）、开源 action tokenizer、全新 VLA 架构（per-layer KV conditioning），以及自适应深度推理变体 MolmoAct2-Think，在 7 个仿真与真实世界基准上全面领先 π₀.₅ 等强基线。MolmoAct2 is a fully open-source Vision-Language-Action (VLA) model that advances its predecessor MolmoAct along five dimensions: a stronger embodied-reasoning VLM backbone Molmo2-ER; three new robot datasets (including the largest open-source bimanual dataset to date, 720 hours); an open-source action tokenizer; a new VLA architecture (per-layer KV conditioning); and an adaptive-depth reasoning variant MolmoAct2-Think, which leads strong baselines such as π₀.₅ across 7 simulation and real-world benchmarks.
当前的机器人基础模型距离真实部署仍有差距：顶级模型是闭源的；开源替代方案受限于昂贵硬件；引入推理的策略推理延迟过高；微调后的成功率仍低于可靠使用的阈值。Current robot foundation models still fall short of real deployment: frontier models are closed; open-weight alternatives are tied to expensive hardware; reasoning-augmented policies incur prohibitive inference latency; and post-fine-tuning success rates remain below the threshold for dependable use.
"Frontier models are closed; open-weight alternatives are tied to expensive hardware; reasoning-augmented policies pay prohibitive latency for their grounding; and fine-tuned success rates remain below the threshold for dependable use."
现有推理增强 VLA（如 MolmoAct、TraceVLA）通过预测深度 token、目标图像或轨迹来提升动作质量，但每步生成大量 token 导致推理延迟过高，无法支持闭环控制。同时，开源机器人数据集分散、质量参差不齐，难以支撑跨具身的多任务学习。Existing reasoning-augmented VLAs (such as MolmoAct and TraceVLA) improve action quality by predicting depth tokens, goal images or trajectories, but generating a large number of tokens at every step drives inference latency too high to support closed-loop control. At the same time, open-source robot datasets are fragmented and uneven in quality, making cross-embodiment multi-task learning hard to support.
MolmoAct2 系统总览。左：从低到中价位三类平台收集、过滤和整理的高质量机器人数据集；中：MolmoAct2 及其自适应深度推理变体 MolmoAct2-Think，通过 per-layer KV conditioning 将 VLM 骨干与 flow-matching 动作专家耦合；右：模型开箱即用地部署于双臂 YAM、SO-100/101 以及 DROID Franka，并适应洗碗、整理、湿实验室自动化、倒茶等现实任务。MolmoAct2 system overview. Left: high-quality robot datasets collected, filtered and curated from three platform tiers spanning low to mid price ranges; Middle: MolmoAct2 and its adaptive-depth reasoning variant MolmoAct2-Think, which couples the VLM backbone with a flow-matching action expert through per-layer KV conditioning; Right: the model is deployed out of the box on the bimanual YAM, SO-100/101 and DROID Franka, adapting to real-world tasks such as dishwashing, tidying, wet-lab automation and pouring tea.
7仿真 + 真实世界基准（最广泛的开源 VLA 评估）simulation and real-world benchmarks (the broadest open-source VLA evaluation)
720hMolmoAct2-BimanualYAM 双臂遥操作数据（迄今最大开源双臂数据集）MolmoAct2-BimanualYAM bimanual teleoperation data (largest open-source bimanual dataset to date)
63.8%Molmo2-ER 在 13 个具身推理基准上的平均分（超越 GPT-5 和 Gemini Robotics ER-1.5）average score of Molmo2-ER across 13 embodied-reasoning benchmarks (ahead of GPT-5 and Gemini Robotics ER-1.5)
98.1%MolmoAct2-Think 在 LIBERO 全套上的平均成功率（当前最优）average success rate of MolmoAct2-Think on the full LIBERO suite (state of the art)
02方法Method
MolmoAct2 在五个维度上推进了前作 MolmoAct：（1）具身推理 VLM 骨干 Molmo2-ER；（2）三个新的开源机器人数据集；（3）开源多具身 action tokenizer OpenFAST；（4）基于 per-layer KV conditioning 的新 VLA 架构；（5）自适应深度推理变体 MolmoAct2-Think。训练分为三个阶段：预训练、后训练（post-training）和部署微调。MolmoAct2 advances its predecessor MolmoAct along five dimensions: (1) the embodied-reasoning VLM backbone Molmo2-ER; (2) three new open-source robot datasets; (3) the open-source multi-embodiment action tokenizer OpenFAST; (4) a new VLA architecture based on per-layer KV conditioning; and (5) the adaptive-depth reasoning variant MolmoAct2-Think. Training is split into three stages: pre-training, post-training and deployment fine-tuning.
Molmo2-ER：具身推理 VLM 骨干Molmo2-ER: an Embodied-Reasoning VLM Backbone
通用 VLM 很少训练机器人策略所需的技能——度量距离、自由空间、跨视角物体跟踪、场景几何。为此，MolmoAct2 基于 Molmo2-4B 进行 specialize-then-rehearse 两阶段训练：General-purpose VLMs are rarely trained on the skills a robot policy needs — metric distance, free space, cross-view object tracking, scene geometry. MolmoAct2 therefore trains on top of Molmo2-4B with a two-stage specialize-then-rehearse recipe:
Stage 1（具身专项化）：在 3.3M 样本的空间-具身语料上微调 20K 步，数据涵盖单图具身 QA、图像 Pointing、视频具身 QA、多图 Ego-Exo 对应推理等六个维度，辅以 8% Tulu-3 文本数据保留语言能力。Stage 1 (embodied specialization): fine-tuning for 20K steps on a 3.3M-sample spatial-embodied corpus that spans six dimensions — single-image embodied QA, image pointing, video embodied QA, multi-image Ego-Exo correspondence reasoning and more — with 8% Tulu-3 text data mixed in to preserve language ability.
Stage 2（联合精化）：继续训练 1.5K 步，将具身语料与 Molmo2 原始多模态中训数据以 p=0.5 的比例混合，序列长度从 4,200 扩展至 16,384。Stage 2 (joint refinement): training continues for a further 1.5K steps, mixing the embodied corpus with Molmo2's original multimodal mid-training data at a ratio of p=0.5, and extending sequence length from 4,200 to 16,384.
Molmo2-ER 在 13 个具身推理基准上平均得分 63.8%，超越其基础模型 Molmo2 17 个百分点，并在 9/13 个基准上超越 Gemini Robotics ER-1.5 Thinking 和 GPT-5。Molmo2-ER scores 63.8% on average across 13 embodied-reasoning benchmarks, exceeding its base model Molmo2 by 17 percentage points, and surpassing Gemini Robotics ER-1.5 Thinking and GPT-5 on 9/13 benchmarks.
MolmoAct2-OpenFAST：开源 Action TokenizerMolmoAct2-OpenFAST: an Open-Source Action Tokenizer
现有 action tokenizer 要么闭源，要么与特定动作空间绑定。MolmoAct2-OpenFAST 是遵循 FAST 方案的开源实现，在五类具身平台的数百万条轨迹上训练，可将 1 秒钟的 32 维连续动作压缩为紧凑的离散序列，支持跨具身的统一 next-token 训练目标。Existing action tokenizers are either closed-source or bound to a particular action space. MolmoAct2-OpenFAST is an open-source implementation following the FAST recipe, trained on millions of trajectories from five embodiment classes; it compresses 1 second of 32-dimensional continuous actions into a compact discrete sequence, supporting a unified cross-embodiment next-token training objective.
MolmoAct2 采用三阶段流水线：预训练阶段将 Molmo2-ER 适配为离散自回归机器人策略；后训练阶段接入 flow-matching 连续动作专家（DiT 风格 transformer），并通过 per-layer KV conditioning 将 VLM 与动作专家耦合——即动作专家的每一层从对应的 VLM 层获取 Key-Value 状态，而非仅使用隐状态。消融实验表明，per-layer KV conditioning 在 LIBERO 全套平均得分 95.9%，优于隐状态条件化（94.0%）和 per-head KV 变体（94.8%）。部署微调阶段在具体具身平台、环境和任务上高效适配。MolmoAct2 uses a three-stage pipeline: the pre-training stage adapts Molmo2-ER into a discrete autoregressive robot policy; the post-training stage attaches a flow-matching continuous action expert (a DiT-style transformer) and couples the VLM with the action expert through per-layer KV conditioning — that is, every layer of the action expert reads Key-Value states from the corresponding VLM layer instead of using hidden states alone. Ablations show that per-layer KV conditioning scores 95.9% on average over the full LIBERO suite, better than hidden-state conditioning (94.0%) and the per-head KV variant (94.8%). The deployment fine-tuning stage then adapts efficiently to a given embodiment, environment and task.
MolmoAct2-Think 概览。在动作生成前，模型预测紧凑的离散深度表示（10×10 网格，128 个深度码值），并通过 per-layer KV conditioning 将其输入动作专家。自适应深度推理：对静态区域复用缓存的深度码，仅对 RGB 发生变化的区域重新预测深度 token，使推理代价与场景变化量成正比而非固定 100 token 的全量预测。MolmoAct2-Think overview. Before generating actions, the model predicts a compact discrete depth representation (a 10×10 grid with 128 depth code values) and feeds it into the action expert through per-layer KV conditioning. Adaptive-depth reasoning: cached depth codes are reused for static regions and depth tokens are re-predicted only where the RGB input changes, making inference cost proportional to the amount of scene change rather than a fixed full prediction of 100 tokens.
MolmoAct2-Think 在推理时维护 10×10 深度码缓冲区，通过余弦相似度（阈值 0.996）逐 patch 比较相邻帧，标记变化区域并选择性重新解码深度 token。这利用了机器人轨迹的时序冗余性，在保留几何空间感知的同时大幅降低延迟。精调时额外引入：10% 深度 token 噪声注入（应对推理时深度不完美预测）和可学习的 per-layer depth gate（从 bias=-4 初始化，逐步学习各层对深度前缀的权重）。At inference time MolmoAct2-Think maintains a 10×10 depth-code buffer, compares adjacent frames patch by patch with cosine similarity (threshold 0.996), marks the changed regions and selectively re-decodes their depth tokens. This exploits the temporal redundancy of robot trajectories, cutting latency substantially while preserving geometric spatial perception. Fine-tuning adds two further ingredients: 10% depth-token noise injection (to cope with imperfect depth prediction at inference) and a learnable per-layer depth gate (initialized from bias=-4, gradually learning how much weight each layer gives the depth prefix).
03实验Experiments
在 7 个仿真和真实世界基准上开展了迄今最广泛的开源 VLA 评估，涵盖 Molmo2-ER 具身推理能力评估、开箱即用部署、高效微调三个维度。The broadest open-source VLA evaluation to date is carried out on 7 simulation and real-world benchmarks, covering three dimensions: the embodied-reasoning ability of Molmo2-ER, out-of-the-box deployment, and efficient fine-tuning.
MolmoAct2 平均成功率 50.1%，超越第二名 OpenVLA-OFT 15 个百分点，在 8 项任务中 7 项排名第一，覆盖静态实验室、厨房、湿实验室、移动操作等场景。MolmoAct2 reaches an average success rate of 50.1%, exceeding the runner-up OpenVLA-OFT by 15 percentage points and ranking first on 7 of the 8 tasks, across static lab, kitchen, wet-lab and mobile-manipulation settings.
MolmoAct2 高效微调综合评估。在 8 项真实世界任务（从化学家准备移液管到将玩具放回货架）上进行系统评估。MolmoAct2 以超出第二名 15% 的大幅优势领先 4 个强基线。Comprehensive evaluation of MolmoAct2 efficient fine-tuning. Systematic evaluation on 8 real-world tasks, from a chemist preparing a pipette to putting toys back on a shelf. MolmoAct2 leads 4 strong baselines by a wide margin of 15% over the runner-up.
RoboEval 基准性能对比。（A）8 项操作任务的逐任务成功率（%），MolmoAct2 在大多数任务上领先，在 Pack Box、Rotate Valve 等长时程任务上尤为突出。（B）跨行为和结果指标的归一化性能雷达图（CT=完成时间，TL=轨迹长度，JPL=关节路径长度，CPL=笛卡尔路径长度，CJ=笛卡尔抖动，JJ=关节抖动，SC=自碰撞，SL=滑移次数）。MolmoAct2 全面领先，成功率 44.3%（超越 π₀.₅ 的 3.8%），且轨迹更短、更稳定。RoboEval benchmark performance comparison. (A) Per-task success rate (%) on 8 manipulation tasks; MolmoAct2 leads on most of them and is especially strong on long-horizon tasks such as Pack Box and Rotate Valve. (B) Radar plot of normalized performance across behavioral and outcome metrics (CT = completion time, TL = trajectory length, JPL = joint path length, CPL = Cartesian path length, CJ = Cartesian jerk, JJ = joint jerk, SC = self-collisions, SL = number of slips). MolmoAct2 leads across the board with a success rate of 44.3% (against 3.8% for π₀.₅), with shorter and more stable trajectories.
MolmoAct2-Think 在空间变化、光照、语言措辞变体和视觉干扰四类 OOD 扰动下，总体成功率 50.69%，超越第二名 OpenVLA-OFT 10.80 个百分点。Under four classes of OOD perturbation — spatial variation, lighting, language phrasing variants and visual distractors — MolmoAct2-Think attains an overall success rate of 50.69%, exceeding the runner-up OpenVLA-OFT by 10.80 percentage points.
推理速度Inference Speed
缓存与 CUDA Graph 优化后的推理控制率。控制率 = 动作 horizon / 端到端延迟；使用 horizon=10，在单张 H100 上于 LIBERO 测量。MolmoAct2 经 CUDA Graph 优化达到 55.79 Hz（原始的 2.42×），MolmoAct2-Think 达到 12.71 Hz（1.58×）。Inference control rate after caching and CUDA Graph optimization. Control rate = action horizon / end-to-end latency; measured on LIBERO with horizon=10 on a single H100. With CUDA Graph optimization MolmoAct2 reaches 55.79 Hz (2.42× the original) and MolmoAct2-Think reaches 12.71 Hz (1.58×).
消融实验Ablations
具身推理骨干：以相同离散动作架构微调，Molmo2-ER 骨干在 LIBERO-Long 上达 83.6%，超越 Molmo2 骨干的 77.6%（+6.0 点）。Embodied-reasoning backbone: fine-tuned with the same discrete-action architecture, the Molmo2-ER backbone reaches 83.6% on LIBERO-Long, beating the 77.6% of the Molmo2 backbone (+6.0 points).
flow 采样数 K：K=8 时平均最优（95.90%），K=1 最弱（94.15%）。Number of flow sampling steps K: K=8 is best on average (95.90%), while K=1 is weakest (94.15%).
精调设计：全量微调 + 离散动作 co-training（无知识隔离）效果最佳（97.20%）；仅微调动作专家为明显失败项（93.05%）。Fine-tuning design: full fine-tuning plus discrete-action co-training (no knowledge isolation) works best (97.20%); tuning the action expert alone is a clear failure case (93.05%).
MolmoAct2-Think 深度精调：混合训练 + 噪声注入 + depth gate 三者组合最优（98.10%），相比无任何增强的基线（97.50%）提升 0.6 点。MolmoAct2-Think depth fine-tuning: combining mixed training, noise injection and the depth gate is best (98.10%), a gain of 0.6 points over the baseline without any of these (97.50%).
04局限性Limitations
Note：以下局限性部分为论文正文中明确陈述（labeled: stated），部分从模型设计中推断（labeled: inferred）。论文未设独立局限性章节。Note: Some of the limitations below are stated explicitly in the body of the paper (labeled: stated), others are inferred from the model design (labeled: inferred). The paper has no dedicated limitations section.
Open 类任务性能较弱（stated）Weak performance on Open-class tasks (stated)
在 MolmoSpaces 基准的 Open（关节体交互）类别上，MolmoAct2-DROID 得分仅 9.5，明显低于 π₀.₅-DROID 的 22.7，表明"articulated-object interaction remains a direction for further improvement"（论文原文）。On the Open (articulated-object interaction) category of the MolmoSpaces benchmark, MolmoAct2-DROID scores only 9.5, clearly below the 22.7 of π₀.₅-DROID, indicating that "articulated-object interaction remains a direction for further improvement" (quoted from the paper).
空间泛化仍是薄弱项（stated）Spatial generalization remains a weak point (stated)
在 OOD 鲁棒性评估中，MolmoAct2 在 Spatial Variance（空间位置超出训练分布）类别下的绝对成功率仅 26.25%，是四类扰动中最低的，表明细粒度空间泛化仍有提升空间（论文指出 "room for improvement on fine-grained spatial generalization"）。In the OOD robustness evaluation, MolmoAct2 attains an absolute success rate of only 26.25% under Spatial Variance (object positions outside the training distribution), the lowest of the four perturbation classes, indicating that fine-grained spatial generalization still has headroom (the paper notes "room for improvement on fine-grained spatial generalization").
MolmoAct2-Think 的 CUDA Graph 加速收益有限（stated）Limited CUDA Graph speedup for MolmoAct2-Think (stated)
MolmoAct2-Think 的自适应深度解码阶段因其自回归特性（序列依赖、变长执行），CUDA Graph 加速仅带来 1.58× 提升，远低于 MolmoAct2 的 2.42×。高延迟（约 12.71 Hz）相对于高频控制需求仍有差距。Because the adaptive-depth decoding stage of MolmoAct2-Think is autoregressive (sequential dependencies, variable-length execution), CUDA Graph delivers only a 1.58× speedup, far below the 2.42× of MolmoAct2. The resulting latency (about 12.71 Hz) still falls short of high-frequency control requirements.
评估平台多样性仍有限（inferred）Evaluation platform diversity is still limited (inferred)
尽管评估覆盖 DROID Franka、SO-100/101、双臂 YAM 三类平台，但均为桌面操作类任务，对移动底盘、人形机器人等更多具身形式的泛化能力尚未验证（从论文范围推断）。Although the evaluation spans three platform classes — DROID Franka, SO-100/101 and the bimanual YAM — all of them are tabletop manipulation tasks, so generalization to further embodiments such as mobile bases or humanoids remains unverified (inferred from the scope of the paper).
MolmoAct2-Think 的深度 token 由 Depth Anything V2 单目估计生成，在反光表面、透明物体或极端光照等场景下深度估计可能失准，进而影响自适应深度推理质量（从数据处理流程推断）。The depth tokens of MolmoAct2-Think are produced by monocular estimation with Depth Anything V2, which can be inaccurate on reflective surfaces, transparent objects or extreme lighting, in turn degrading the quality of adaptive-depth reasoning (inferred from the data-processing pipeline).
