CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

01 动机Motivation

现有 VLA 模型将视觉语言模型（VLM）直接用于动作预测，通常通过动作量化或简单回归头来输出控制指令，忽略了机器人动作预测与语言 token 生成之间的本质差异，导致泛化能力和精度受限。Existing VLA models apply vision-language models (VLM) directly to action prediction, usually emitting control commands through action quantization or a simple regression head. They ignore the essential difference between robot action prediction and language token generation, which limits generalization and precision.

"Rather than naively adapting pretrained VLMs for action prediction through simple action quantization or adding regression heads, we propose a componentized VLA architecture with specialized action modules."

现有方法存在两大问题：第一，动作空间是连续的高维多模态分布，离散化会丢失精度；第二，视觉语言模型的 next-token 预测范式并不适合时序动作序列建模。CogACT 的核心动机在于"认知"（理解场景与指令）和"行动"（输出精确控制序列）应由专门模块分别承担。Existing methods suffer from two major problems. First, the action space is a continuous, high-dimensional, multimodal distribution, and discretization loses precision. Second, the next-token prediction paradigm of vision-language models is not well suited to modeling temporal action sequences. The core motivation of CogACT is that "cognition" (understanding the scene and the instruction) and "action" (producing precise control sequences) should be carried out by separate, dedicated modules.

CogACT 结果对比 — 图1：CogACT 在多个机器人平台（Google robot、WidowX、Realman、Franka）上的成功率对比，以及 DiT 动作模块规模扩展曲线和真实机器人任务执行示例（顺序叠杯、拾放未见物体）。Figure 1: Success-rate comparison of CogACT across several robot platforms (Google robot, WidowX, Realman, Franka), together with the scaling curve of the DiT action module and examples of real-robot task execution (sequential cup stacking, pick-and-place of unseen objects).

74.8%Google Robot 仿真平均成功率
（vs OpenVLA 34.3%）Average success rate on Google Robot simulation
(vs OpenVLA 34.3%)

71.2%Realman 真实机器人平均成功率
（vs OpenVLA 12.1%）Average success rate on the Realman real robot
(vs OpenVLA 12.1%)

61.4%Franka 真实机器人平均成功率
（vs OpenVLA 6.8%）Average success rate on the Franka real robot
(vs OpenVLA 6.8%)

>35%仿真中相对 OpenVLA 的成功率提升幅度Relative success-rate gain over OpenVLA in simulation

02 方法Method

CogACT 将整体模型分为三个组件：视觉模块（Vision Module）、语言模块（Language Module）和扩散动作模块（Diffusion Action Module）。前两者复用成熟的预训练 VLM，后者为专为机器人控制设计的 Diffusion Transformer。CogACT splits the overall model into three components: a Vision Module, a Language Module and a Diffusion Action Module. The first two reuse a mature pretrained VLM, while the last is a Diffusion Transformer designed specifically for robot control.

CogACT 架构图 — 图2：CogACT 组件化架构。视觉模块（DINOv2 + SigLIP）将观测图像编码为 256 个 visual token；语言模块（LLaMA-2）融合视觉与语言指令，输出 cognition feature；扩散动作模块以 cognition feature 为条件，通过 DiT 预测多步动作序列。Figure 2: The componentized architecture of CogACT. The vision module (DINOv2 + SigLIP) encodes the observed image into 256 visual tokens; the language module (LLaMA-2) fuses the visual and language instructions and outputs a cognition feature; the diffusion action module, conditioned on that cognition feature, predicts a multi-step action sequence with a DiT.

视觉与语言模块（Cognition）Vision and Language Modules (Cognition)

视觉模块采用 DINOv2 和 SigLIP 双编码器，将当前图像编码为 256 个 visual token，通过可学习的投影层输入至语言模块。语言模块基于 LLaMA-2，接收视觉 token 和自然语言任务指令，输出 cognition feature——一个压缩了场景理解和任务目标的特征向量，作为后续动作生成的条件信号。The vision module uses the dual encoders DINOv2 and SigLIP to encode the current image into 256 visual tokens, which are fed into the language module through a learnable projection layer. The language module is built on LLaMA-2; it takes the visual tokens together with the natural-language task instruction and outputs a cognition feature—a feature vector that compresses scene understanding and the task goal, serving as the conditioning signal for the subsequent action generation.

扩散动作模块（Diffusion Action Module）Diffusion Action Module

动作模块基于 Diffusion Transformer（DiT），以 cognition feature 为条件，在扩散过程中预测 7-DoF 末端执行器动作序列（3D 平移、旋转和夹爪状态），每次预测涵盖 16 个时间步的未来动作。消融实验显示预测步长为 15 时综合性能最优（平均 62.5%），过长（31 步）反而下降至 51.2%。DiT 在等参数规模下显著优于 MLP 基线（DiT-Base 89M：62.5% vs MLP 7-Layer 89M：52.5%）。The action module is based on a Diffusion Transformer (DiT). Conditioned on the cognition feature, it predicts a 7-DoF end-effector action sequence (3D translation, rotation and gripper state) through the diffusion process, each prediction covering 16 future time steps. Ablations show that a prediction horizon of 15 yields the best overall performance (62.5% on average), whereas an overly long horizon (31 steps) drops to 51.2%. At matched parameter scale, DiT clearly outperforms the MLP baseline (DiT-Base 89M: 62.5% vs MLP 7-Layer 89M: 52.5%).

自适应动作融合策略 — 图3：Adaptive Action Ensemble 策略（K=2 示例）。通过计算历史预测与当前预测的 cosine similarity，对历史预测加权融合，避免多模态动作分布下的均值退化问题。相比 Action Chunking（50.7%）和 Temporal Ensemble（58.9%），Adaptive Ensemble 在平均成功率上达到 62.5%。Figure 3: The Adaptive Action Ensemble strategy (illustrated with K=2). By computing the cosine similarity between the historical predictions and the current prediction, the historical predictions are fused with weights, avoiding the mean-degeneration problem under a multimodal action distribution. Compared with Action Chunking (50.7%) and Temporal Ensemble (58.9%), Adaptive Ensemble reaches an average success rate of 62.5%.

Adaptive Action Ensemble

执行时，模型维护最近 K 帧的历史动作预测。对于每个新时间步，以当前预测与历史预测的 cosine similarity 为权重，加权融合历史预测与当前输出，作为最终执行动作。这一策略有效解决了多模态动作分布下直接均值会产生非法动作的问题，同时保留了时序一致性带来的平滑性提升。At execution time, the model maintains the historical action predictions of the most recent K frames. For each new time step, it uses the cosine similarity between the current prediction and the historical predictions as weights, fusing the historical predictions with the current output into the final executed action. This strategy effectively resolves the problem that a plain average under a multimodal action distribution yields invalid actions, while keeping the smoothness gained from temporal consistency.

动作模块规模扩展 — 图4：动作模块规模与仿真平均成功率的关系。从 DiT-Small（13M, 58.5%）到 DiT-Base（89M, 62.5%）再到 DiT-Large（308M, 64.8%），呈现对数线性扩展规律，表明 CogACT 的动作模块具有良好的 scaling 特性。Figure 4: Relationship between action-module scale and average success rate in simulation. From DiT-Small (13M, 58.5%) to DiT-Base (89M, 62.5%) and further to DiT-Large (308M, 64.8%), a log-linear scaling trend emerges, indicating that the action module of CogACT has favorable scaling properties.

03 实验Experiments

在 Open X-Embodiment 数据集的 25 个子集（共 22.5M 帧）上训练，使用 SIMPLER 仿真框架评估 Google Robot 和 WidowX，并在 Realman 和 Franka 真实机器人上进行零样本迁移测试，对比基线包括 RT-1、RT-1-X、RT-2-X、Octo 系列和 OpenVLA。Training is carried out on 25 subsets of the Open X-Embodiment dataset (22.5M frames in total). The SIMPLER simulation framework is used to evaluate Google Robot and WidowX, and zero-shot transfer is tested on the Realman and Franka real robots. The baselines compared against include RT-1, RT-1-X, RT-2-X, the Octo series and OpenVLA.

仿真评测：Google Robot（SIMPLER）Simulation Evaluation: Google Robot (SIMPLER)

方法Method	Pick Coke Can	Move Near	Open/Close Drawer	Open Top Drawer	平均 (VM)Average (VM)
RT-2-X	78.7	77.9	25.0	3.7	46.3
OpenVLA	18.0	56.3	63.0	0.0	34.3
CogACT (Ours)	91.3	85.0	71.8	50.9	74.8

仿真评测：WidowX Robot（SIMPLER）Simulation Evaluation: WidowX Robot (SIMPLER)

方法Method	Put Spoon on Towel	Put Carrot on Plate	Stack Green Block	Put Eggplant in Basket	平均 (VM)Average (VM)
Octo-Small	41.7	8.2	0.0	56.7	26.7
OpenVLA	4.2	0.0	0.0	12.5	4.2
CogACT (Ours)	71.7	50.8	15.0	67.5	51.3

真实机器人：Realman RobotReal Robot: Realman Robot

方法Method	Pick（均值）Pick (mean)	Stack（均值）Stack (mean)	Place（均值）Place (mean)	整体平均Overall average
Octo-Base	8.3	0.0	12.5	4.9
OpenVLA	8.3	15.6	12.5	12.1
CogACT (Ours)	70.8	82.3	60.4	71.2

泛化评测：Realman Robot（未见物体属性）Generalization Evaluation: Realman Robot (Unseen Object Attributes)

方法Method	Unseen Colors	Unseen Shapes	Unseen Categories	平均Average
OpenVLA	0.0	6.3	12.5	6.3
CogACT (Ours)	87.5	81.3	25.0	64.6

真实机器人：Franka RobotReal Robot: Franka Robot

方法Method	Close Oven	Open Oven	Pick Bowl	Pick Brush	平均Average
OpenVLA	18.2	0.0	9.1	0.0	6.8
CogACT (Ours)	63.6	72.7	72.7	36.4	61.4

真实机器人实验环境 — 图：Realman 机器人（左）和 Franka 机器人（右）的真实实验环境。Realman 包含 Pick、Stack、Place 三类任务；Franka 包含 Close Oven、Open Oven、Pick Bowl、Pick Brush 四类任务。Figure: The real experimental setups of the Realman robot (left) and the Franka robot (right). Realman covers three task families: Pick, Stack and Place; Franka covers four task families: Close Oven, Open Oven, Pick Bowl and Pick Brush.

消融实验Ablation Studies

动作架构对比中，DiT-Small（13M）的平均成功率（58.5%）已超越 MLP 7-Layer（89M，52.5%），体现出 DiT 在动作序列建模上的参数效率优势。多步预测步长消融实验中，预测 15 步（62.5%）优于单步（42.8%）和 3 步（55.5%），说明适当的时序窗口对提升动作连续性至关重要。动作融合策略对比：Adaptive Ensemble（62.5%）> Temporal Ensemble（58.9%）> Action Chunking（50.7%）。In the action-architecture comparison, the average success rate of DiT-Small (13M) (58.5%) already surpasses MLP 7-Layer (89M, 52.5%), showing the parameter efficiency of DiT for action-sequence modeling. In the ablation on the multi-step prediction horizon, predicting 15 steps (62.5%) beats single-step prediction (42.8%) and 3 steps (55.5%), indicating that an appropriate temporal window is critical for improving action continuity. Comparison of action-fusion strategies: Adaptive Ensemble (62.5%) > Temporal Ensemble (58.9%) > Action Chunking (50.7%).

04 局限性Limitations

Note: 论文未设立专门的 Limitations 章节。以下局限性部分来自作者在实验描述中的明确说明（标注为"作者说明"），部分由设计决策推断（标注为"推断"）。The paper has no dedicated Limitations section. The limitations below come partly from explicit statements by the authors in the experimental description (marked "author statement") and partly from inferences drawn from design decisions (marked "inferred").

动作空间仅支持 7-DoF 单臂末端执行器控制（作者说明）The action space supports only 7-DoF single-arm end-effector control (author statement)

当前模型专注于"7 degrees of freedom (DoF) in this work"的夹爪控制，不支持双臂、全身控制等更复杂的机器人形态。训练数据也限定为"single-arm end-effector control and at least one third-person camera perspective"的数据集。The current model focuses on gripper control with "7 degrees of freedom (DoF) in this work" and does not support more complex robot embodiments such as bimanual or whole-body control. The training data is likewise restricted to datasets with "single-arm end-effector control and at least one third-person camera perspective".

排除了部分数据集，限制了训练数据覆盖（作者说明）Some datasets were excluded, limiting the coverage of the training data (author statement)

作者明确排除了 Language Table 和 Droid 数据集，原因是"their significant distribution disparities with other data"。这意味着模型在类似这些数据集的任务分布上可能表现欠佳。The authors explicitly excluded the Language Table and Droid datasets because of "their significant distribution disparities with other data". This implies that the model may underperform on task distributions similar to those datasets.

未见类别的泛化能力有限（作者说明）Limited generalization to unseen categories (author statement)

在 Realman 机器人的未见属性泛化测试中，Unseen Categories 成功率仅 25.0%，远低于 Unseen Colors（87.5%）和 Unseen Shapes（81.3%），说明模型对完全新颖的物体类别泛化能力有待提升。In the unseen-attribute generalization test on the Realman robot, the success rate on Unseen Categories is only 25.0%, far below Unseen Colors (87.5%) and Unseen Shapes (81.3%), showing that the model's generalization to entirely novel object categories still needs improvement.

真实部署需要任务相关的微调数据（推断）Real-world deployment requires task-specific fine-tuning data (inferred)

各真实机器人平台需要收集 48–400 条任务演示数据进行微调，限制了零样本泛化到全新任务的能力。这是当前数据驱动 VLA 方法的共性瓶颈，并非 CogACT 独有。Each real-robot platform requires collecting 48–400 task demonstrations for fine-tuning, which limits zero-shot generalization to entirely new tasks. This is a common bottleneck of current data-driven VLA methods rather than something unique to CogACT.