CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models

01 动机Motivation

当前主流 VLA 模型直接将语言指令与视觉观测映射到机器人动作，缺乏中间推理过程。这与人类"先在脑海中规划目标状态，再执行"的认知方式相悖，也限制了模型在复杂长视野任务中的泛化能力。Mainstream VLA models map language instructions and visual observations directly to robot actions, with no intermediate reasoning process. This runs counter to the human cognitive pattern of "first planning the goal state in the mind, then executing", and it also limits generalization on complex, long-horizon tasks.

"We propose to incorporate explicit visual chain-of-thought (CoT) reasoning into VLAs by predicting future image frames autoregressively as visual goals before generating a short action sequence to achieve these goals."

CoT-VLA vs Vanilla VLA 对比 — **图 1 · 框架对比。**传统 VLA（左）直接从当前观测与语言指令预测动作；CoT-VLA（右）首先以自回归方式生成未来子目标图像作为视觉中间推理步骤，然后基于该子目标生成动作序列。CoT-VLA 可同时利用带动作标注的机器人演示数据与无动作标注的视频数据进行训练。**Figure 1 · Framework comparison.** A vanilla VLA (left) predicts actions directly from the current observation and the language instruction; CoT-VLA (right) first generates a future sub-goal image autoregressively as a visual intermediate reasoning step, and then generates the action sequence conditioned on that sub-goal. CoT-VLA can be trained on action-labeled robot demonstrations and action-free video data alike.

+17%真实机器人操作任务 vs SOTAReal-robot manipulation tasks vs SOTA

+6%LIBERO 仿真 benchmarkLIBERO simulation benchmark

7BVILA-U 骨干参数量VILA-U backbone parameter count

46.7%预训练阶段带来的相对提升（Franka-Tabletop）Relative gain from the pre-training stage (Franka-Tabletop)

核心洞察在于：若模型能生成"接下来 n 步后的场景应该是什么样"的图像，则这一预测本身就构成了对任务进度的显式规划。无动作标注的视频数据（如 EPIC-KITCHEN、Something-Something V2）同样包含丰富的视觉动态先验，可用于训练子目标图像生成能力，从而扩大有效训练数据规模。The core insight: if a model can generate an image of "what the scene should look like n steps from now", that prediction itself constitutes explicit planning over task progress. Action-free video data (e.g. EPIC-KITCHEN, Something-Something V2) likewise carries rich visual dynamics priors and can be used to train sub-goal image generation, thereby enlarging the effective training corpus.

02 方法Method

CoT-VLA 以 VILA-U 为基础模型，采用"两阶段条件生成"：先以自回归 Transformer 生成子目标图像 token，再以 full-attention 并行解码 action chunk。训练分三阶段：通用预训练 → 机器人预训练（含无动作视频）→ 任务微调。CoT-VLA takes VILA-U as its base model and adopts a "two-stage conditional generation" scheme: an autoregressive Transformer first generates the sub-goal image tokens, then a full-attention pass decodes the action chunk in parallel. Training proceeds in three stages: general pre-training → robot pre-training (including action-free video) → task fine-tuning.

CoT-VLA 架构总览 — **图 2 · 架构总览。**模型以 VILA-U 为骨干，输入当前图像观测与语言指令，先通过 causal attention 自回归生成子目标图像（256 个 RQ-VAE token，分辨率 256×256），再以 full attention 并行预测 m=10 步动作 chunk。闭环部署时，每执行完一个 action chunk 即重新感知并生成新子目标。**Figure 2 · Architecture overview.** With VILA-U as backbone, the model takes the current image observation and the language instruction, first generates the sub-goal image autoregressively under causal attention (256 RQ-VAE tokens at 256×256 resolution), and then predicts an m=10-step action chunk in parallel under full attention. At closed-loop deployment, the robot re-perceives and generates a new sub-goal after each action chunk is executed.

VILA-U 与残差量化图像 tokenVILA-U and residual-quantized image tokens

VILA-U 是一个统一的多模态基础模型，同时支持图像理解与生成。图像通过 RQ-VAE（Residual Quantization VAE）编码为 16×16×4 的 token 网格（残差深度为 4），在 256×256 分辨率下每帧共 256 个 token。深度 Transformer 逐层预测残差 token，从而在离散 token 空间内实现高质量图像生成。动作 token 则直接拼接在图像/文本序列之后，以统一的 Transformer 解码。VILA-U is a unified multimodal foundation model that supports both image understanding and image generation. Images are encoded by an RQ-VAE (Residual Quantization VAE) into a 16×16×4 token grid (residual depth 4), giving 256 tokens per frame at 256×256 resolution. A depth Transformer predicts the residual tokens level by level, achieving high-quality image generation within a discrete token space. Action tokens are appended directly after the image/text sequence and decoded by the same unified Transformer.

Hybrid Attention：视觉生成用 Causal，动作预测用 FullHybrid Attention: Causal for visual generation, Full for action prediction

混合注意力机制示意图 — **图 3 · Hybrid Attention 机制。**生成图像与文本 token 时使用 causal（因果）注意力以保证自回归一致性；预测动作 token 时切换为 full attention，使整个 action chunk 内每个时间步的动作都能相互感知，实现更平滑的 action chunking。特殊分隔 token 用于区分两种注意力模式。**Figure 3 · The hybrid attention mechanism.** Causal attention is used when generating image and text tokens, so as to preserve autoregressive consistency; for action tokens the model switches to full attention, letting the actions at every timestep within an action chunk attend to one another and yielding smoother action chunking. A special separator token distinguishes the two attention modes.

这一设计将自回归图像生成与并行动作解码统一在同一 Transformer 中，避免了双网络带来的额外复杂度。Action chunking（m=10）减少了闭环控制的决策频率，提升执行流畅度。This design unifies autoregressive image generation and parallel action decoding inside a single Transformer, avoiding the extra complexity of a two-network setup. Action chunking (m=10) lowers the decision frequency of closed-loop control and makes execution smoother.

三阶段训练流程Three-stage training pipeline

阶段一：通用预训练Stage 1: General pre-training

使用图文对、视频数据训练 VILA-U 的多模态理解与生成能力Train VILA-U's multimodal understanding and generation ability on image-text pairs and video data
奠定视觉语言基础，无需机器人数据Lays the vision-language foundation; no robot data required

阶段二：机器人预训练Stage 2: Robot pre-training

在 Open X-Embodiment 数据集上同时训练子目标图像生成与动作预测Jointly train sub-goal image generation and action prediction on the Open X-Embodiment dataset
引入无动作标注的视频数据（EPIC-KITCHEN、Something-Something V2），扩大视觉动态先验Bring in action-free video data (EPIC-KITCHEN, Something-Something V2) to broaden the visual dynamics prior
子目标时间间隔 n 从数据集特定范围均匀采样（如 Bridge: n∈[5,10]，TOTO: n∈[20,24]）The sub-goal time interval n is sampled uniformly from a dataset-specific range (e.g. Bridge: n∈[5,10], TOTO: n∈[20,24])

阶段三：目标任务微调Stage 3: Target-task fine-tuning

在目标任务的少量演示数据上微调，保持子目标图像生成头与动作预测头的联合训练Fine-tune on a small set of target-task demonstrations, keeping the sub-goal image generation head and the action prediction head jointly trained
Franka-Tabletop 实验表明此阶段相比直接微调带来 46.7% 相对提升（53.7% → 78.8%）Franka-Tabletop experiments show that this stage brings a 46.7% relative gain over direct fine-tuning (53.7% → 78.8%)

03 实验Experiments

实验在两个平台展开：LIBERO 仿真 benchmark（4 个任务分组）与真实 Franka-Tabletop 机器人平台（6 个操作任务，含 3 个单指令与 3 个多指令任务）。同时在 Bridge-V2 数据集上评估跨任务泛化。基线包括 Diffusion Policy、Octo 和 OpenVLA。Experiments span two platforms: the LIBERO simulation benchmark (4 task suites) and a real Franka-Tabletop robot platform (6 manipulation tasks, comprising 3 single-instruction and 3 multi-instruction tasks). Cross-task generalization is additionally evaluated on the Bridge-V2 dataset. Baselines include Diffusion Policy, Octo and OpenVLA.

LIBERO Benchmark（仿真）LIBERO Benchmark (simulation)

模型Model	Spatial	Object	Goal	Long	平均Average
Diffusion Policy	78.3±1.1%	92.5±0.7%	68.3±1.2%	50.5±1.3%	72.4±0.7%
Octo (fine-tuned)	78.9±1.0%	85.7±0.9%	84.6±0.9%	51.1±1.3%	75.1±0.6%
OpenVLA (fine-tuned)	84.7±0.9%	88.4±0.8%	79.2±1.0%	53.7±1.3%	76.5±0.6%
CoT-VLA-7B（本文）CoT-VLA-7B (ours)	87.5±1.4%	91.6±0.5%	87.6±0.6%	69.0±0.8%	81.13±0.6%

CoT-VLA 在所有 4 个 LIBERO 子分组上均取得最优，尤其在长视野任务 LIBERO-Long 上提升显著（53.7% → 69.0%），体现了视觉子目标对长时序规划的帮助。CoT-VLA is best on all 4 LIBERO suites, with a particularly clear gain on the long-horizon LIBERO-Long suite (53.7% → 69.0%), showing how visual sub-goals help long-horizon planning.

Bridge-V2 跨任务泛化Bridge-V2 cross-task generalization

模型Model	Visual	Motion	Semantic	Language
SUSIE	30%	10%	20%	40%
Octo	35%	10%	0%	40%
OpenVLA	75%	45%	40%	75%
CoT-VLA（本文）CoT-VLA (ours)	65%	60%	50%	70%

在 Bridge-V2 上，CoT-VLA 在 Motion 泛化（60% vs 45%）与 Semantic 泛化（50% vs 40%）上超越 OpenVLA，但在 Visual 与 Language 分类上略低于 OpenVLA（65% vs 75%，70% vs 75%）。On Bridge-V2, CoT-VLA beats OpenVLA on Motion generalization (60% vs 45%) and Semantic generalization (50% vs 40%), but is slightly behind OpenVLA on the Visual and Language categories (65% vs 75%, 70% vs 75%).

真实机器人 Franka-TabletopReal robot: Franka-Tabletop

Franka-Tabletop 真实机器人实验结果 — **图 4 · Franka-Tabletop 真实机器人实验。**6 个操作任务（含 3 个单指令窄领域任务与 3 个多指令宽领域任务）的成功率对比。CoT-VLA 在平均成功率上超越所有基线 17%。**Figure 4 · Franka-Tabletop real-robot experiments.** Success-rate comparison over 6 manipulation tasks (3 narrow-domain single-instruction tasks and 3 broad-domain multi-instruction tasks). CoT-VLA exceeds every baseline by 17% in average success rate.

定性结果Qualitative results

CoT-VLA 定性执行示例 — **图 5 · 定性执行示例。**展示 CoT-VLA 在多个真实操作任务中，生成子目标图像与实际执行轨迹的对应关系。预测的子目标图像在视觉上接近任务中间状态，引导机器人准确完成操作。**Figure 5 · Qualitative rollout examples.** The correspondence between generated sub-goal images and the actual execution trajectories across several real manipulation tasks. The predicted sub-goal images are visually close to the intermediate task states and guide the robot to complete the manipulation accurately.

消融实验Ablation studies

消融实验结果 — **图 6 · 消融实验。**在 LIBERO-Spatial 与 LIBERO-Goal 上，逐步添加各组件（action chunking、hybrid attention、visual CoT）均带来稳定提升。Franka-Tabletop 预训练消融显示：预训练阶段带来 46.7% 相对提升（53.7% → 78.8%）。**Figure 6 · Ablation studies.** On LIBERO-Spatial and LIBERO-Goal, adding each component in turn (action chunking, hybrid attention, visual CoT) brings a consistent gain. The Franka-Tabletop pre-training ablation shows that the pre-training stage brings a 46.7% relative gain (53.7% → 78.8%).

消融实验验证了每个组件的独立贡献：Action chunking 提供稳定基础；Hybrid attention 使动作预测更连贯；Visual CoT 带来最终的性能跃升，尤其在需要多步规划的任务中效果最为明显。The ablations verify the independent contribution of each component: action chunking provides a stable base; hybrid attention makes action prediction more coherent; visual CoT delivers the final performance jump, most clearly on tasks that require multi-step planning.

04 局限性Limitations

注：以下局限性均为论文作者在 Section 5 明确陈述（stated by the authors）。Note: All limitations below are explicitly stated by the authors in Section 5.

推理速度慢——约 7× 延迟开销Slow inference — roughly 7× latency overhead

论文明确指出："Our method requires generating 256 image tokens before action tokens, leading to a 7×slowdown on average with an action chunk size of 10."。在实时控制场景下，推理延迟是主要瓶颈，限制了对高频控制任务的适用性。The paper states plainly: "Our method requires generating 256 image tokens before action tokens, leading to a 7×slowdown on average with an action chunk size of 10." In real-time control settings, inference latency is the main bottleneck and limits applicability to high-frequency control tasks.

自回归图像生成质量低于扩散模型Autoregressive image generation quality trails diffusion models

VILA-U 的自回归图像生成在视觉细节上弱于扩散模型（如 DALL-E、Stable Diffusion）。子目标图像存在模糊或细节丢失，可能在需要精密视觉对准的任务中降低动作质量。论文在 OOD 泛化实验中也发现，使用生成子目标（20%/0%）远低于使用真实子目标（60%/40%）的成功率，印证了图像质量的瓶颈。VILA-U's autoregressive image generation is weaker in visual detail than diffusion models (e.g. DALL-E, Stable Diffusion). Sub-goal images can be blurry or lose detail, which may degrade action quality on tasks that require precise visual alignment. In the OOD generalization experiments the paper also finds that success rates with generated sub-goals (20%/0%) are far below those with ground-truth sub-goals (60%/40%), confirming the image-quality bottleneck.

Action chunking 引入动作不连续性Action chunking introduces action discontinuities

预测固定长度的动作序列（chunk）会在两个 chunk 交界处出现动作不连续，且缺乏逐步反馈调整能力。当任务需要精细的实时感知-动作闭环时，chunk 边界的抖动可能影响执行稳定性。Predicting a fixed-length action sequence (chunk) produces action discontinuities at the boundary between two chunks, and offers no step-by-step feedback correction. When a task requires a fine-grained real-time perception-action loop, jitter at chunk boundaries may affect execution stability.

对全新任务的泛化能力受计算约束限制Generalization to entirely new tasks is limited by compute constraints

论文指出，模型在面对训练分布之外的全新视觉推理任务时泛化能力有限，部分原因在于计算约束（模型规模、生成 token 数）限制了推理深度与多样性。The paper notes that the model generalizes only to a limited degree to entirely new visual reasoning tasks outside the training distribution, partly because compute constraints (model size, number of generated tokens) restrict the depth and diversity of reasoning.