MM-ACT: Learn from Multimodal Parallel Generation to Act
统一文本、图像、动作的 VLA 模型，通过跨模态并行生成提升机器人操控能力A VLA model that unifies text, images and actions, improving robotic manipulation through cross-modal parallel generation
MM-ACT 将文本、图像与动作纳入共享 token 空间，用 Context-Shared Multimodal Learning 统一训练三种模态的生成，动作解码只需一步并行推断，真实机器人操控成功率达 72.0%，超越 π₀（70.0%）。MM-ACT places text, images and actions in a shared token space and jointly trains the generation of all three modalities with Context-Shared Multimodal Learning. Action decoding needs only a single parallel inference step, and the success rate on real-robot manipulation reaches 72.0%, surpassing π₀ (70.0%).
通用机器人策略需要同时具备语义理解（任务规划）与环境交互（动作预测）两种能力，但现有方法往往将二者分离处理，难以充分利用多模态信号之间的互补关系。A generalist robotic policy must possess both semantic understanding (task planning) and environment interaction (action prediction), yet existing methods usually handle the two separately and fail to exploit the complementarity among multimodal signals.
"A generalist robotic policy needs both semantic understanding for task planning and the ability to interact with the environment through predictive capabilities."
图 1：不同统一 VLA 范式对比。MM-ACT（右）将文本、图像与动作三种模态统一在共享 token 空间中，通过 Context-Shared Multimodal Learning 同时监督三种模态的生成，而早期方法（左、中）仅支持单一或两种模态的联合训练。Figure 1: Comparison of unified VLA paradigms. MM-ACT (right) unifies the text, image and action modalities in a shared token space and supervises the generation of all three simultaneously through Context-Shared Multimodal Learning, whereas earlier methods (left, middle) support joint training of only one or two modalities.
+9.25%跨模态学习带来的额外增益Additional gain from cross-modal learning
现有方法的不足Shortcomings of Existing Methods
单模态辅助训练：许多 VLA 仅将视觉或语言生成作为辅助任务，缺乏统一的多模态生成框架。Unimodal auxiliary training: many VLAs use only visual or language generation as an auxiliary task and lack a unified multimodal generation framework.
跨模态上下文未充分利用：文本描述与图像预测对动作生成的潜在增益未被系统挖掘。Under-exploited cross-modal context: the potential gain that textual descriptions and image predictions bring to action generation has not been systematically explored.
02方法 MethodMethod
MM-ACT 将文本 token、图像 token 与动作 token 编码进同一序列，通过 re-mask 并行解码生成文本/图像，通过一步并行解码生成动作，并用 Context-Shared Multimodal Learning 在共享上下文下统一监督三种模态。MM-ACT encodes text tokens, image tokens and action tokens into a single sequence, generates text and images by re-mask parallel decoding and actions by one-step parallel decoding, and supervises all three modalities under a shared context with Context-Shared Multimodal Learning.
图 2：MM-ACT 架构。输入由多视角观测图像、任务语言指令、文本描述及机器人状态（可选）组成的 modality-interleaved token sequence；通过 bidirectional attention 的 masked token prediction，同时预测文本、图像与动作三种模态的 token。Figure 2: The MM-ACT architecture. The input is a modality-interleaved token sequence composed of multi-view observation images, the task language instruction, the text description and the robot state (optional); masked token prediction with bidirectional attention predicts the tokens of the text, image and action modalities simultaneously.
统一 Token 空间Unified Token Space
模型为文本、图像、动作分别配置模态专属的 tokenizer，将三种模态映射到同一维度的 token 序列。推理时，所有已知信息（观测图像、语言指令、文本描述、机器人状态）拼接为共享上下文，掩码位置对应待生成的目标 token。The model gives text, images and actions their own modality-specific tokenizers, mapping the three modalities into token sequences of the same dimension. At inference, all known information (observation images, language instruction, text description, robot state) is concatenated into a shared context, and the masked positions correspond to the target tokens to be generated.
Re-mask 并行解码（文本与图像）Re-mask Parallel Decoding (Text and Image)
对于文本和图像生成，MM-ACT 采用 re-mask parallel decoding 策略：每个解码步骤中，模型并行预测所有掩码位置的 token，然后依置信度保留高分 token，低置信度 token 重新被掩码（re-masked）并在下一步重新预测。文本使用线性调度；图像使用余弦调度控制每步揭示的 token 比例。For text and image generation, MM-ACT adopts a re-mask parallel decoding strategy: at every decoding step the model predicts the tokens at all masked positions in parallel, keeps the high-confidence tokens by score, and re-masks the low-confidence tokens so that they are predicted again at the next step. Text uses a linear schedule; images use a cosine schedule to control the fraction of tokens revealed per step.
一步并行解码（动作）One-Step Parallel Decoding (Actions)
动作生成采用 one-step parallel decoding，即"predict all masked tokens simultaneously"，单次前向传播即可输出完整动作 chunk，支持 40 Hz 的实时控制频率，推理耗时仅约 0.22–0.23 秒（chunk size 8–16）。Action generation adopts one-step parallel decoding, i.e. "predict all masked tokens simultaneously": a single forward pass outputs a complete action chunk, supporting a 40 Hz real-time control frequency, with an inference cost of only about 0.22–0.23 s (chunk size 8–16).
图 3：两阶段训练流程。Stage 1 仅训练文本与图像生成（固定动作模态权重为 0）；Stage 2 引入动作生成，辅助模态权重约为 0.05–0.1。共享上下文（shared context）在两阶段均保持一致，确保跨模态信号互相增强。Figure 3: The two-stage training pipeline. Stage 1 trains text and image generation only (the action modality weight is fixed to 0); Stage 2 introduces action generation, with auxiliary modality weights of about 0.05–0.1. The shared context stays identical across both stages, ensuring that cross-modal signals reinforce each other.
Context-Shared Multimodal Learning
MM-ACT 的核心训练范式：三种模态的生成任务共享完全相同的输入上下文，使用统一的 cross-entropy loss 同时监督文本、图像与动作的 token 预测。这使模型在学习动作时能同时接收文本语义与图像预测的梯度信号，形成双向增强。The core training paradigm of MM-ACT: the generation tasks of the three modalities share exactly the same input context and are supervised together by a unified cross-entropy loss over text, image and action token prediction. While learning actions the model therefore also receives gradient signals from text semantics and image prediction, forming a bidirectional reinforcement.
03实验 ExperimentsExperiments
在三个基准上评估：LIBERO（仿真，in-domain）、Franka 真实机器人（3 个任务）、RoboTwin2.0（仿真，out-of-domain，8 个双臂任务）。基线方法包括 π₀、UniVLA、OpenVLA、OpenVLA-OFT。Evaluation on three benchmarks: LIBERO (simulation, in-domain), a real Franka robot (3 tasks) and RoboTwin2.0 (simulation, out-of-domain, 8 bimanual tasks). Baselines include π₀, UniVLA, OpenVLA and OpenVLA-OFT.
主要结果对比Main Results
方法Method
LIBERO 平均LIBERO Avg.
Franka 真实Franka Real
RoboTwin2.0
OpenVLA
76.5%
—
—
OpenVLA-OFT
—
58.6%
23.13%
π₀
94.2%
70.0%
48.13%
UniVLA
95.5%
—
—
MM-ACT（本文）MM-ACT (ours)
96.3%
72.0%
52.38%
图 4：MM-ACT 在 RoboTwin 未见（unseen）环境中生成的场景图像可视化。模型能够在新场景下合理预测未来帧，体现出跨模态学习赋予的泛化能力。图像质量：PSNR 14.23、SSIM 0.80、LPIPS 0.09。Figure 4: Visualization of scene images generated by MM-ACT in unseen RoboTwin environments. The model predicts plausible future frames in novel scenes, reflecting the generalization ability conferred by cross-modal learning. Image quality: PSNR 14.23, SSIM 0.80, LPIPS 0.09.
LIBERO 子任务细节LIBERO Sub-task Breakdown
LIBERO 子集LIBERO Suite
MM-ACT 成功率MM-ACT Success Rate
LIBERO-Spatial
97.8%
LIBERO-Object
99.4%
LIBERO-Goal
94.8%
LIBERO-Long
88.0%
平均Average
96.3%
消融实验 AblationsAblations
作者从三个维度进行消融（均在 RoboTwin2.0 上测试）：The authors ablate along three dimensions (all tested on RoboTwin2.0):
动作解码策略：chunk size=16 时，re-mask PD 达到 56.75%（+13.00% vs one-step PD 43.75%），但推理时间为 1.06s vs 0.23s；最终选择 one-step PD 以兼顾速度与精度。Action decoding strategy: with chunk size=16, re-mask PD reaches 56.75% (+13.00% vs one-step PD at 43.75%), but inference time is 1.06s vs 0.23s; one-step PD is finally chosen to balance speed and accuracy.
文本/图像解码策略：图像使用 re-mask 解码可带来 +5.62%（48.75% vs 43.13%）增益；文本使用 re-mask 带来 +3.37%（46.50%）。Text / image decoding strategy: re-mask decoding for images brings a gain of +5.62% (48.75% vs 43.13%); re-mask for text brings +3.37% (46.50%).
机器人状态输入：加入 robot state 对图像模态有 +8.37%（51.50%）增益，但对文本模态略有负面影响。最终方案结合图像+状态输入达到 52.38%。Robot state input: adding the robot state yields a +8.37% (51.50%) gain for the image modality, but is slightly detrimental to the text modality. The final scheme, combining image and state input, reaches 52.38%.
Stage 2 引入动作模态后，文本生成精度从 81.5% 下降至 68.7%，作者认为"text modality is prone to overfitting with increased training steps"，而图像模态则持续受益于跨模态学习。After Stage 2 introduces the action modality, text generation accuracy drops from 81.5% to 68.7%; the authors argue that "text modality is prone to overfitting with increased training steps", while the image modality keeps benefiting from cross-modal learning.
04局限性 LimitationsLimitations
注意：原论文无独立的 Limitations 章节。以下各点：标注为「论文已陈述」者来自正文明确描述；标注为「推断」者为基于方法设计的合理推断。Note: the original paper has no standalone Limitations section. Among the points below, those marked "stated in the paper" come from explicit descriptions in the text, while those marked "inferred" are reasonable inferences based on the method design.
文本生成精度在 Stage 2 退化（论文已陈述）Text generation accuracy degrades in Stage 2 (stated in the paper)
加入动作模态训练后，文本任务规划准确率从 81.5% 降至 68.7%。作者解释为"text modality is prone to overfitting with increased training steps"，但并未给出针对性的缓解方案。这意味着模型在语言理解与动作执行之间存在训练权衡。After the action modality is added to training, text task-planning accuracy falls from 81.5% to 68.7%. The authors explain this as "text modality is prone to overfitting with increased training steps", but give no targeted mitigation. This implies a training trade-off between language understanding and action execution.
re-mask 解码的速度-精度权衡（论文已陈述）Speed-accuracy trade-off of re-mask decoding (stated in the paper)
消融实验表明，chunk size=16 时 re-mask PD 比 one-step PD 精度高 13%，但推理时间从 0.23s 增至 1.06s，无法满足 40 Hz 实时控制需求。当前方案选择 one-step PD 牺牲精度换速度，高频控制场景下的最优解码策略仍待探索。Ablations show that with chunk size=16 re-mask PD is 13% more accurate than one-step PD, but inference time grows from 0.23s to 1.06s, which cannot meet the 40 Hz real-time control requirement. The current scheme picks one-step PD, trading accuracy for speed; the optimal decoding strategy for high-frequency control remains to be explored.
RoboTwin2.0 整体成功率偏低（推断）Overall RoboTwin2.0 success rate is low (inferred)
最优配置下 RoboTwin2.0 的成功率为 52.38%，在 8 个双臂任务中仍有近半数失败。双臂协同操作的难度远超单臂场景，当前框架对双臂时序协调建模的能力有待进一步验证。Under the best configuration the RoboTwin2.0 success rate is 52.38%, and nearly half of the 8 bimanual tasks still fail. Bimanual coordinated manipulation is far harder than single-arm settings, and the current framework's ability to model temporal coordination between the two arms needs further validation.
图像生成质量（推断）Image generation quality (inferred)
Stage 2 图像质量指标 PSNR 14.23、SSIM 0.80、LPIPS 0.09，PSNR 数值相对偏低，说明生成图像与真实未来帧之间仍存在较大像素级差异，影响其作为预测辅助信号的可靠性。The Stage 2 image quality metrics are PSNR 14.23, SSIM 0.80 and LPIPS 0.09; the relatively low PSNR indicates a still-large pixel-level gap between the generated images and the true future frames, which limits their reliability as an auxiliary predictive signal.
