MolmoAct: Action Reasoning Models that can Reason in Space
将空间推理引入机器人基础模型——从深度感知到轨迹规划再到精准动作Bringing spatial reasoning into robot foundation models — from depth perception to trajectory planning to precise action
Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Li, Bohan Fang, Jieyu Zhang, Yi Ru Wang, Sangho Lee, Winson Han, Wilbert Pumacay, Angelica Wu, Rose Hendrix, Karen Farley, Eli VanderBilt, Ali Farhadi, Dieter Fox, Ranjay Krishna · UW / Allen Institute for AI
MolmoAct 提出了一类名为 Action Reasoning Model (ARM) 的机器人基础模型，通过三阶段结构化流水线——深度感知 token 编码、中层空间轨迹规划、低层精准动作预测——在不牺牲端到端可训练性的前提下，赋予机器人「在空间中推理」的能力。模型在零样本 SimplerEnv 测试中达到 70.5% 成功率，并在真实场景操作中全面超越 π₀-FAST。MolmoAct introduces a class of robot foundation models named Action Reasoning Model (ARM). Through a three-stage structured pipeline — depth perception token encoding, mid-level spatial trajectory planning, and low-level precise action prediction — it endows robots with the ability to "reason in space" without sacrificing end-to-end trainability. The model reaches a 70.5% success rate on zero-shot SimplerEnv and comprehensively surpasses π₀-FAST in real-world manipulation.
当前大多数视觉-语言-动作（VLA）模型将感知与指令直接映射到控制信号，缺乏中间推理步骤，导致适应性、泛化性和语义可解释性受限。论文指出，大语言模型从 chain-of-thought 推理中获益匪浅，而机器人学领域却鲜有类似结构。Most current vision-language-action (VLA) models map perception and instructions directly to control signals, lacking any intermediate reasoning step, which limits adaptability, generalization and semantic interpretability. The paper points out that large language models benefit greatly from chain-of-thought reasoning, whereas robotics has had little of an analogous structure.
"Reasoning is central to purposeful action, yet most robotic foundation models map perception and instructions directly to control, which limits adaptability, generalization, and semantic grounding."
作者认为，机器人的推理应当「根植于空间理解」而非纯语言推理——轨迹、深度信息与物理空间才是机器人决策的真正基础。MolmoAct 的核心贡献是将这一理念落地：在自回归生成的同一 token 序列中，依次推断深度、轨迹与动作，实现可解释、可干预的行为。The authors argue that robot reasoning should be "grounded in spatial understanding" rather than purely linguistic — trajectories, depth information and physical space are the real basis of robot decision-making. The core contribution of MolmoAct puts this idea into practice: within one autoregressively generated token sequence it successively infers depth, trajectory and action, yielding interpretable and steerable behavior.
Figure 1：MolmoAct 系统概览。模型接收 RGB 观测与自然语言指令，依次输出深度 token（深度感知）、2D end-effector 轨迹 trace（空间规划）与低层机器人动作 token（精准控制）。三个推理链相互条件化，共同支撑可解释、可干预的操作行为。Figure 1: Overview of the MolmoAct system. The model takes an RGB observation and a natural-language instruction, then successively outputs depth tokens (depth perception), a 2D end-effector trajectory trace (spatial planning) and low-level robot action tokens (precise control). The three reasoning chains condition on one another and jointly support interpretable, steerable manipulation behavior.
70.5%SimplerEnv zero-shot Visual Matching Avg
86.6%LIBERO 平均成功率 （4 类任务）LIBERO average success rate (4 task suites)
+22.7%双臂操作超越 π₀-FASTBimanual manipulation gain over π₀-FAST
MolmoAct 采用自回归 Transformer 架构，将深度估计、轨迹生成与动作预测统一在同一 token 序列中，按顺序条件化：深度 token 由图像与指令确定，轨迹 trace 以深度为条件，动作 token 则同时以深度与轨迹为条件。MolmoAct adopts an autoregressive Transformer architecture that unifies depth estimation, trajectory generation and action prediction in a single token sequence, conditioned in order: depth tokens are determined by the image and the instruction, the trajectory trace is conditioned on depth, and action tokens are conditioned on both depth and trajectory.
Figure 2：MolmoAct 模型结构与训练流程。左侧展示三阶段解码结构（深度 → 轨迹 → 动作）；右侧为预训练（网络数据 + 推理数据）与后训练（机器人数据微调）两阶段的数据混合方案。预训练使用 26.3M 样本，包含 38.7% 动作推理数据、38.7% 轨迹条件数据与 21.5% 网络数据。Figure 2: MolmoAct model architecture and training pipeline. Left: the three-stage decoding structure (depth → trajectory → action). Right: the data mixture for the two stages, pre-training (web data + reasoning data) and post-training (fine-tuning on robot data). Pre-training uses 26.3M samples, comprising 38.7% action reasoning data, 38.7% trajectory-conditioned data and 21.5% web data.
Stage 1 · Depth Perception Tokens
利用预训练深度估计器 Depth Anything V2 从 RGB 图像提取深度图，再经过 VQVAE（codebook 维度 128，序列长度 100 tokens）离散化编码。100 个深度 token 被插入到文本 token 序列中，令模型获得「3D understanding, which is critical for robotic manipulation」。VQVAE 在 1000 万张桌面操作深度图上训练，分辨率 320×320。A pre-trained depth estimator, Depth Anything V2, extracts a depth map from the RGB image, which is then discretized by a VQVAE (codebook dimension 128, sequence length 100 tokens). The 100 depth tokens are inserted into the text token sequence, giving the model "3D understanding, which is critical for robotic manipulation". The VQVAE is trained on ten million tabletop manipulation depth maps at 320×320 resolution.
Stage 2 · Visual Reasoning Trace
模型预测 end-effector 在图像平面上的未来轨迹，表示为折线（1–5 个关键点），以像素坐标直接叠加在观测图像上。与纯语言规划不同，trajectory trace 是 2D 空间中「future motion of the end effector」的直接表达，可被人类实时查看和编辑，实现行为的可干预性（steerability）。The model predicts the future trajectory of the end-effector on the image plane, represented as a polyline (1–5 keypoints) overlaid directly on the observation in pixel coordinates. Unlike purely linguistic planning, the trajectory trace is a direct expression of the "future motion of the end effector" in 2D space, and can be inspected and edited by a human in real time, giving the behavior steerability.
Stage 3 · Action Token Tokenization
机器人动作以每维 256 个均匀宽度 bin 离散化。与随机分配词汇 token 不同，作者将 Qwen2 tokenizer 最后 256 个 token（字节级 BPE 符号）单调递增地分配给各 bin，从而保留动作的序数结构——相邻 bin 对应相邻符号——为优化提供「smoother starting point」。此外，采用 similarity-preserving initialization 初始化动作 token 嵌入，将预训练效率提升超 5×（相比 GR00T N1.5 的 50,000 GPU 小时，MolmoAct 仅需 9,728 小时）。Robot actions are discretized into 256 uniform-width bins per dimension. Instead of assigning vocabulary tokens at random, the authors map the last 256 tokens of the Qwen2 tokenizer (byte-level BPE symbols) monotonically onto the bins, thereby preserving the ordinal structure of actions — adjacent bins correspond to adjacent symbols — and providing a "smoother starting point" for optimization. In addition, similarity-preserving initialization of the action token embeddings improves pre-training efficiency by more than 5× (against the 50,000 GPU hours of GR00T N1.5, MolmoAct needs only 9,728 hours).
训练策略Training Strategy
预训练（256 × H100，100k 步，batch 512）在网络多模态数据与推理数据上进行，建立空间理解与语言理解的基础；中训练（128 × H100，50k 步，batch 128）在 MolmoAct Dataset（10,689 条 Franka 单臂轨迹，93 类家庭任务，平均 112 timesteps/条）上强化操作能力；后训练在目标任务数据上微调。Pre-training (256 × H100, 100k steps, batch 512) runs on web multimodal data and reasoning data, establishing the foundation for spatial and language understanding; mid-training (128 × H100, 50k steps, batch 128) strengthens manipulation ability on the MolmoAct Dataset (10,689 single-arm Franka trajectories, 93 household task categories, 112 timesteps per trajectory on average); post-training fine-tunes on target-task data.
03实验Experiments
评估覆盖仿真基准（SimplerEnv、LIBERO）和真实世界单臂/双臂操作任务，基线包括 π₀-FAST、GR00T N1.5、SpatialVLA、ThinkAct 等主流 VLA 模型。Evaluation covers simulation benchmarks (SimplerEnv, LIBERO) and real-world single-arm / bimanual manipulation tasks, with baselines including mainstream VLA models such as π₀-FAST, GR00T N1.5, SpatialVLA and ThinkAct.
SimplerEnv（仿真，Google Robot）SimplerEnv (simulation, Google Robot)
MolmoAct 在长视野任务（LIBERO-Long）上超越 ThinkAct 6.3 个百分点，体现了空间推理在复杂序列操作中的优势。值得注意：π₀-FAST 在 Spatial 和 Object 子任务上仍优于 MolmoAct。On long-horizon tasks (LIBERO-Long) MolmoAct exceeds ThinkAct by 6.3 percentage points, showing the advantage of spatial reasoning in complex sequential manipulation. Worth noting: π₀-FAST still outperforms MolmoAct on the Spatial and Object suites.
真实世界操作（Real-World）Real-World Manipulation
Figure 5：真实场景单臂与双臂任务中各模型的 task progression 曲线（带误差棒）。MolmoAct 在单臂任务超越 π₀-FAST 约 10%，在双臂操作任务超越 π₀-FAST 22.7%，体现了轨迹推理对复杂协作操作的重要性。Figure 5: Task progression curves (with error bars) for each model on real-world single-arm and bimanual tasks. MolmoAct exceeds π₀-FAST by about 10% on single-arm tasks and by 22.7% on bimanual manipulation, showing the importance of trajectory reasoning for complex coordinated manipulation.Figure 6a：真实场景 out-of-distribution 泛化能力对比。MolmoAct 在 OOD 设置下平均提升 23.3%，对新背景、新物体姿态、新指令的适应性显著强于基线。Figure 6a: Comparison of out-of-distribution generalization in real-world settings. MolmoAct improves by 23.3% on average under OOD conditions, adapting to new backgrounds, new object poses and new instructions markedly better than the baselines.
可操纵性（Steerability）与指令泛化Steerability and Instruction Generalization
Figure 9：可操纵性评估结果。基于轨迹 trace 的人工干预成功率达 75%，显著高于纯语言引导（33%）。左图为不同引导方式的成功率柱状图；右图展示轨迹叠加在摄像头视图上的可视化效果，便于操作者实时理解和修正机器人意图。Figure 9: Steerability evaluation results. Human intervention based on the trajectory trace reaches a 75% success rate, far above purely linguistic guidance (33%). Left: a bar chart of success rates under different guidance modes. Right: a visualization of the trajectory overlaid on the camera view, letting the operator understand and correct the robot's intent in real time.
人类评估中，MolmoAct 在指令跟随 Elo 评分上排名第一，对战 SpatialVLA 胜率 58%；轨迹引导的 steerability 成功率 75%，比语言引导高出 42 个百分点。MolmoAct Dataset 中训练为通用性能带来平均 5.5% 的提升。In human evaluation MolmoAct ranks first on instruction-following Elo, with a 58% win rate against SpatialVLA; trajectory-guided steerability reaches a 75% success rate, 42 percentage points above language guidance. Mid-training on the MolmoAct Dataset brings an average 5.5% gain in general performance.
消融实验Ablations
消融研究验证了三个核心设计选择的有效性：（1）深度 token 的引入对需要 3D 理解的操作任务至关重要；（2）trajectory trace 作为中层规划的引入显著提升了长视野任务成功率和可操纵性；（3）action tokenization 中 similarity-preserving initialization 对预训练效率的提升超过 5×（9,728 vs. GR00T N1.5 的 50,000 GPU 小时）。MolmoAct Dataset 的中训练阶段为现实场景操作带来平均 5.5% 的性能增益。The ablation study verifies the effectiveness of three core design choices: (1) depth tokens are essential for manipulation tasks that require 3D understanding; (2) introducing the trajectory trace as mid-level planning markedly raises long-horizon success rate and steerability; (3) similarity-preserving initialization in action tokenization improves pre-training efficiency by more than 5× (9,728 vs. 50,000 GPU hours for GR00T N1.5). The mid-training stage on the MolmoAct Dataset yields an average 5.5% performance gain in real-world manipulation.
04局限性Limitations
说明：原论文的 Appendix G「Limitations and Potential Solutions」在 arXiv HTML 渲染中未完整呈现。以下局限性综合了论文正文的明确说明（标注「论文陈述」）与从模型设计中推断的问题（标注「推断」）。Note: Appendix G, "Limitations and Potential Solutions", of the original paper is not fully rendered in the arXiv HTML. The limitations below combine points explicitly stated in the paper body (marked "stated in the paper") with issues inferred from the model design (marked "inferred").
深度估计的前置依赖（论文陈述）Upstream dependence on depth estimation (stated in the paper)
MolmoAct 依赖 Depth Anything V2 提供深度先验，VQVAE 在 1000 万张桌面操作场景上训练。若部署环境与训练分布差异过大（如户外场景、透明/反光物体），深度估计质量可能下降，进而影响整个推理链的可靠性。MolmoAct relies on Depth Anything V2 for its depth prior, and the VQVAE is trained on ten million tabletop manipulation scenes. If the deployment environment differs too much from the training distribution (outdoor scenes, transparent or reflective objects), depth estimation quality may degrade and in turn undermine the reliability of the whole reasoning chain.
2D 轨迹 trace 的表达局限（推断）Expressive limits of the 2D trajectory trace (inferred)
空间规划仅以图像平面 2D 折线表示，无法直接编码三维末端执行器姿态或与环境的接触力信息。对于需要精确 6-DoF 控制或力控的操作任务，当前的 trace 表示可能不足以充分约束动作预测。Spatial planning is expressed only as a 2D polyline on the image plane and cannot directly encode the 3D end-effector pose or contact forces with the environment. For manipulation tasks that require precise 6-DoF control or force control, the current trace representation may not constrain action prediction sufficiently.
数据规模与任务多样性受限（推断）Limited data scale and task diversity (inferred)
MolmoAct Dataset 由五名操作员在两个月内采集，共 10,689 条轨迹、93 类家庭任务，仅覆盖单臂 Franka 机器人。相比大规模开源机器人数据集（如 Open X-Embodiment），数据规模与平台多样性仍有差距，跨具身迁移能力有待进一步验证。The MolmoAct Dataset was collected by five operators over two months, totalling 10,689 trajectories across 93 household task categories, and covers only a single-arm Franka robot. Compared with large-scale open-source robot datasets such as Open X-Embodiment, its data scale and platform diversity still lag behind, and cross-embodiment transfer remains to be verified.
训练计算成本仍然较高（推断）Training compute cost remains high (inferred)
尽管相比 GR00T N1.5（50,000 GPU 小时）已大幅降低，MolmoAct 的预训练仍需 9,728 GPU 小时（256 × H100），中训练需额外 2,304 GPU 小时（128 × H100），对大多数学术机构和中小型团队而言仍是较高门槛。Although greatly reduced compared with GR00T N1.5 (50,000 GPU hours), pre-training MolmoAct still takes 9,728 GPU hours (256 × H100) and mid-training a further 2,304 GPU hours (128 × H100), which remains a high barrier for most academic institutions and small or mid-sized teams.
